This is an old revision of the document!
Table of Contents
Intro: Minimal CentOS 6.5
The Spectrum XMC1151 digitizer libraries we have require a specific flavour of Linux to operate.
Download the specific version by getting the http://vault.centos.org/6.5/isos/x86_64/CentOS-6.5-x86_64-minimal.iso which is the “minimal” CentOS 6.5.
- Actual direct download link for minimal 6.5 CentOS ISO: http://archive.kernel.org/centos-vault/6.5/isos/x86_64/CentOS-6.5-x86_64-minimal.iso (md5sum is 0d9dc37b5dd4befa1c440d2174e88a87 CentOS-6.5-x86_64-minimal.iso)
Kernel from OS install 2016.04.14
Linux lvisf 2.6.32-431.el6.x86_64 #1 SMP Fri Nov 22 03:15:09 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux
Base CentOS
Using the 500_00746_r100_SDR7000_Linux_SG.pdf document, set up the core data system.
DVD Boot
NOTE: Press the Delete key while booting in order to change the boot to either the USB PLEXTOR drive or the LEXAR CFAST drive
- BIOS → Boot → Boot Option #1
- Then F4: Save & Exit
- Attached the external USB CDROM drive to the computer via the USB3 port (USB stick with UNETBOOTIN did NOT WORK for me?!) (UnetBootin will work with later version of CentOS image… 6.5 is broken)
- Insert the Minimal CentOS 6.5 Install Disk
- Boot and set the BIOS to boot off the Plextor drive (our USB based DVD drive)
- At CentOS 6 Welcome, Click → Next
- Language: English (English)
- Keyboard: U.S. English
- Storage Devices: Basic Storage Devices
- NOTE: If another OS detected, select: Fresh Installation (SURE YOU WANT TO DELETE THAT OS!?)
- Hostname: lvisf
- Timezone: New York
- root Password: the usual
- Partitioning: Use All Space
- Select the drive (for this case, the 60 GB (or 60000 MB drive) and put that to the right, then Write Changes
- Wait for install to complete (205 packages in ~4 minutes)
- Drive will eject
- Remove DVD
- Reboot
CentOS Configure
lvisf Config
Network Temporary Configuration (DHCP)
- Setup the system for DHCP for now
- Log in as root
- Modify /etc/sysconfig/network-scripts/ifcfg-eth0 so
ONBOOT=yes
- Startup network (will take a moment, DHCP)
sudo ifup eth0
Prerequisites for Compiling Kernel Modules
- http://www.tldp.org/LDP/lkmpg/2.6/html/x121.html ← build kernel module link for an example
- Install the development tools
yum groupinstall 'Development Tools'
- Get some useful programs
yum install nano -y yum install wget -y
- Grab the ethernet driver while you still have access
- Current driver is here: https://downloadmirror.intel.com/13663/eng/igb-5.3.5.3.tar.gz
mkdir ~/down cd ~/down wget https://downloadmirror.intel.com/13663/eng/igb-5.3.5.3.tar.gz
Ethernet Driver Install and Configuration
- Log in as root on the main console since ethernet will be disabled by default
- Install Intel ethernet drivers [Only needed if compiling on Kontron flight hardware]:
- Download igb-5.3.4.4.tar.gz to a USB drive
- Current driver is here: https://downloadmirror.intel.com/13663/eng/igb-5.3.5.3.tar.gz
- Uncompress in the home directory
tar xvf igb-5.3.4.4.tar.gz
- Build and install the drvier
cd ~/tmp/igb-5.3.4.4/src/ make sudo make install
- Modify this module file
sudo nano /etc/sysconfig/modules/lvisf.modules
- Put the following into this file
#!/bin/sh # start intel network driver /sbin/modprobe igb
- Change permissions so this script runs at boot time
sudo chmod 755 /etc/sysconfig/modules/lvisf.modules
- Configure our network device
- Change the eth0 definition
- editing the config file
vi /etc/sysconfig/network-scripts/ifcfg-eth0
- to the following
DEVICE=eth0 HWADDR=00:80:82:1B:E7:04 TYPE=Ethernet UUID=98bd88d8-8fe8-4b8e-bca0-a58bd3306717 ONBOOT=yes NM_CONTROLLED=yes BOOTPROTO=static NAME="System eth0" IPADDR=10.4.1.2 NETMASK=255.255.255.0
- Add our default gateway
- Edit the network configuration
vi /etc/sysconfig/network
- End up looking like this
NETWORKING=yes HOSTNAME=lvisf NETWORKING_IPV6=no IPV6INIT=no GATEWAY=10.4.1.1
- Ensure our name server can be found
- Edit resolv.conf
vi /etc/resolv.conf
- End up with this
search gsfc.nasa.gov nameserver 128.183.10.134 nameserver 128.183.50.17
- Turn off IPV6 (found here: https://www.centos.org/forums/viewtopic.php?t=7989)
sudo sysctl -w net.ipv6.conf.default.disable_ipv6=1 sudo sysctl -w net.ipv6.conf.all.disable_ipv6=1
- Change the eth1 definition
- editing the config file
vi /etc/sysconfig/network-scripts/ifcfg-eth1
- to the following
DEVICE=eth1 HWADDR=00:80:82:1B:E7:05 TYPE=Ethernet UUID=5eb0ea0f-b1e8-43ab-a64d-83d3d9e1dc95 ONBOOT=yes NM_CONTROLLED=yes BOOTPROTO=static NAME="System eth1" IPADDR=192.168.0.2 NETMASK=255.255.255.0
- Change the eth1:1 definition
- editing the config file
vi /etc/sysconfig/network-scripts/ifcfg-eth1:1
- to the following
DEVICE=eth1:1 HWADDR=00:80:82:1B:E7:05 TYPE=Ethernet UUID=5eb0ea0f-b1e8-43ab-a64d-83d3d9e1dc95 ONBOOT=yes NM_CONTROLLED=yes BOOTPROTO=static NAME="System eth1:1" IPADDR=129.100.0.2 NETMASK=255.255.255.0
- REMOVE the keyboard and mouse and never use them again (USB interrupts)
- Add lvis user to the system (still as root (logged in via SSH for me now))
- Add the user
useradd lvis
- Set password
passwd lvis
- Add lvis user to the list of sudo users
- Modify the super user file with the following (as logged in as root)
visudo
- After the line that has root and ALL and ALL permissions, insert this line
lvis ALL=(ALL) ALL
File System Extras
- Add ntfs drive capability
- Add the epel repository
sudo yum install epel-release
- Add ntfs to our system
sudo yum install fuse fuse-ntfs-3g
- Add Exfat drive capability
- Add the epel repository and this Nux Dextop repository
sudo yum install epel-release sudo rpm -Uvh http://li.nux.ro/download/nux/dextop/el6/x86_64/nux-dextop-release-0-2.el6.nux.noarch.rpm
- Add Exfat to our system
sudo yum install exfat-utils fuse-exfat
- An example mounting an Exfat system is
sudo mount -t exfat /dev/sdc1 /mnt/tmp
X11
- This will allow you to run X11
# core X11 sudo yum install xorg-x11-xauth xorg-x11-fonts-* xorg-x11-utils xterm -y # required for tuna sudo yum install xauth pygtk2-libglade -y
- Install tuna tuning package
sudo yum install tuna -y
Remove Unused Services
- Remove exim (NOTE, this was not on the CentOS Minimal I just installed 2018.01.04 (but leaving for historical))
- Stop the service
sudo service exim stop
- Remove the package from the system
sudo rpm -e --nodeps exim
- Remove auditd
sudo service auditd stop # now sudo chkconfig auditd off # after reboot
- Remove postfix
- Stop the service
sudo service postfix stop
- Remove the package from the system
sudo rpm -e --nodeps postfix
Spectrum Config
Still as root logged into the main desktop
- Disable the Firewall
- Turn off the firewall
sudo service iptables stop sudo service ip6tables stop
- Make this permanent
sudo chkconfig iptables off sudo chkconfig ip6tables off
- Disable SELinux
- Edit the configuration file
sudo vi /etc/selinux/config
- Modify the file
- From
SELINUX=enforcing
- To
SELINUX=disabled
- Reboot for all changes to take place
Minimize Kernel Modules
- Disable ipv6, this addition is required along with a line in the modprobe file below
- Modifiy /etc/sysconfig/network
sudo vi /etc/sysconfig/network
- And append the following two lines
NETWORKING_IPV6=no IPV6INIT=no
- Add these two lines to sshd_config
sudo nano /etc/ssh/sshd_config # lvisf - ipv4 only AddressFamily inet ListenAddress 0.0.0.0
- Blacklist kernel modules we don't require
- Edit a file lvisf-blacklist.conf
sudo vi /etc/modprobe.d/lvisf-blacklist.conf
- Add the following lines
# lvisf kernel module blacklist # version 1.0 - dlr - 20160418 # ipv6 blacklist ipv6 # NMI watchdog blacklist iTCO_wdt blacklist iTCO_vendor_support blacklist i2c_i801 # blacklist igb blacklist lpc_ich blacklist shpchp blacklist dm_mirror # intel sound blacklist soundcore blacklist snd blacklist snd_seq blacklist snd_hda_intel blacklist snd_hda_codec_hdmi # USB controller (still loads) blacklist xhci_hcd # this is required to REMOVE ipv6 module, I am not sure why # http://www.cyberciti.biz/faq/redhat-centos-disable-ipv6-networking/ install ipv6 /bin/true
Extras
- These two commands will remove some other modules I couldn't figure out how to make permanently go away
# USB driver... if you're using the nousb kernel, this will work sudo modprobe --remove xhci_hcd sudo modprobe --remove dm_mirror
Minimal Modules
Module Size Used by e1000 170646 0 sg 29350 0 ext4 374902 3 jbd2 93427 1 ext4 mbcache 8193 1 ext4 sd_mod 39069 3 crc_t10dif 1541 1 sd_mod ahci 42215 2 i915 624511 1 drm_kms_helper 44321 1 i915 drm 280012 2 i915,drm_kms_helper i2c_algo_bit 5935 1 i915 i2c_core 31084 4 i915,drm_kms_helper,drm,i2c_algo_bit video 20674 1 i915 output 2409 1 video dm_mod 84209 9
lvisf code
SVN Repository
NOTE: This is obsolete (the repository link), but left here for posterity.
- Dave Yancich sent me this note back in May:
Repository URL: https://zj.yancich.com:8443/svn/NASA/trunk/lvisF username "lvis" password: standard lvis password This is a cmake project. To rebuild the make files: In "build" directory type "cmake ../" Then "make" to build everything. "make clean" to clean the programs and kernel modules.
Build Document
(Yancich sent this to me in March 2016) Original document is configuration_and_build_procedures_for_lvisf_and_support_libraries.docx
Requirements
- Install cmake
sudo yum install cmake -y
- Install htop
sudo yum install htop -y
- Install jed
sudo yum install ftp://ftp.pbone.net/mirror/centos.karan.org/el5/extras/testing/i386/RPMS/jed-0.99.18-5.el5.kb.i386.rpm -y
- Install nano
sudo yum install nano -y
- Install p7zip and pigz
sudo yum install p7zip pigz -y
- Install PCI utilities
sudo yum install pciutils -y
- Install screen
sudo yum install screen -y
- Install telnet
sudo yum install telnet -y
- Install wget
sudo yum install wget -y
- Install the development package
sudo yum groupinstall "Development Tools"
- Install development libraries for md5sum and gzip in real time
sudo yum install openssl-devel -y
- Install md5deep so we can check file integrity
sudo yum install md5deep -y
- Install man pages
sudo yum install man -y sudo yum install man-pages -y
- Install kernel development source
cd mkdir download cd ~/download wget http://vault.centos.org/6.5/os/x86_64/Packages/kernel-devel-2.6.32-431.el6.x86_64.rpm sudo yum --nogpgcheck localinstall ~/download/kernel-devel-2.6.32-431.el6.x86_64.rpm -y
Code Base
- Create the repoistory home and check out the source code
- Create a repository home in /opt
sudo mkdir /opt/lvisf
- Set proper permissions
sudo chown -R lvis:lvis /opt/lvisf
- [Optional] Uncompress our locally attached mount with snapshot (faster than network initial checkout)
cd / tar jxvf /media/LVIS-GH-2TB_WD209/lvisf_build/lvisf_sourcesnapshot_20160816_svn439.tar.bz2
- [Either] Check out he raw source code from scratch
cd /opt/lvisf svn co https://svn.yancich.com:8443/svn/NASA/LVIS/trunk/ src
- [Or] Update it if you used the tar above
cd /opt/lvisf/src svn update
- Enter the user home directory and create symbolic links
cd /home/lvis mkdir src ln -s /opt/lvisf/src/* /home/lvis/src/
- Add the following lines to the .bashrc for the lvis user
- Edit the file
nano /home/lvis/.bash_profile
- Add these lines
# For SVN commit comment editor export SVN_EDITOR=nano # For the Innovative digitizer export WD_BASEDIR=/home/lvis/src/Innovative/WinDriver # For the SyncClock timer card export PLX_SDK_DIR=/home/lvis/src/PLX/PlxSdk # For the spectrum digitizer export SSP_INSTALL_ROOT=/home/lvis/src/spectrum
- Edit the file
sudo nano /etc/ld.so.conf.d/lvisf.conf
- Add these lines
/home/lvis/src/Innovative/winCloudDrvr/winclouddrvr_installation/lib /home/lvis/src/MIC3753/linux_driver_source_3.2.7.0_64bit/libs /home/lvis/src/spectrum/shared/linux/pc64/lib
- Reload the library path
sudo ldconfig
SyncClock32 PLX Drivers
- Build all drivers
- Go to the source code home directory
cd /home/lvis/src/PLX/PlxSdk/Driver
- Execute the build script
./buildalldrivers
- Build the samples
- Go to the source directory
cd /home/lvis/src/PLX/PlxSdk
- Make the programs
make
- Load the driver
- To the binary directory
cd /home/lvis/src/PLX/PlxSdk/Bin
- Execute the script for the 9030 card (our syncclock32)
sudo ./Plx_load 9030 # to just see what boards are supported, run the script with no argument ./Plx_load
MIC 3753 DIO Drivers
NOTE These drivers NEED to be built as root, you cannot sudo to build these drivers
- Become root all the time
sudo bash
- Go to the code base directory
cd /home/lvis/src/MIC3753/linux_driver_source_3.2.7.0_64bit/drivers/driver_base/src/lnx_ko
- Build the drivers
make
- Go into our specific card directory
cd /home/lvis/src/MIC3753/linux_driver_source_3.2.7.0_64bit/drivers/pci1753_mic3753_pcm3753i/src/lnx_ko
- Build the drivers
# to disable IRQs in this driver, just use this code where the IRQ calls are commented out cp init.c.no_irq init.c make
- Load the drivers (this crashes when i load the bio1753 (IRQ!!! issue)
cd /home/lvis/src/MIC3753/linux_driver_source_3.2.7.0_64bit/drivers/bin insmod biokernbase.ko insmod bio1753.ko
- Exit the sudo bash
exit
Innovative Digitizer Drivers
Innovative OEM Drivers
- Make a directory for the Innovative code
sudo mkdir /usr/Innovative
- Copy base Innovative support software into /usr
sudo cp /home/lvis/src/INNO-RAW/* /usr/Innovative/
- Fix the permssions so we can execute this script
sudo chmod 755 /usr/Innovative/Setup.sh
- Create this link to the kernel source so the Innovative driver modules can build
sudo ln -s /usr/src/kernels/2.6.32-431.el6.x86_64 /usr/src/linux
- Move into and run the setup script
cd /usr/Innovative sudo ./Setup.sh
Innovative Custom Drivers
- Build the wdapi support library
- Enter the code directory
cd /home/lvis/src/Innovative/WinDriver/src/wdapi
- Compile the library
make -C LINUX
- Create this symbolic link, but the SVN checkout already has it
ln -s /home/lvis/src/Innovative/WinDriver/src/wdapi/LINUX/libwdapi.so /home/lvis/src/Innovative/winCloudDrvr/winclouddrvr_installation/lib/libwdapi1180.so
- Build the winCloudDrvr
- Enter the source directory
cd /home/lvis/src/Innovative/winCloudDrvr
- Compile
make -C linux
- Build our custom driver
- Enter the directory
cd /home/lvis/src/Innovative/winCloudDrvr/winclouddrvr_installation/redist
- Configure and compile
./configure --disable-usb-support make
- Install
sudo make install
To load the driver:
- insmod our custom driver
sudo /opt/lvisf/src/Innovative/winCloudDrvr/winclouddrvr_installation/redist/wdreg winCloudDrvr auto
Spectrum Digitizer Drivers
- Create the symbolic link for /spectrum
sudo ln -s /opt/lvisf/src/spectrum /spectrum
- Enter the code directory
cd /home/lvis/src/spectrum/shared/linux/pc64/bin
- Copy the drivers for our kernel into the binary directory
cp -a /home/lvis/src/spectrum/shared/linux/pc64/bin/`uname -r`/*.ko /home/lvis/src/spectrum/shared/linux/pc64/bin/
- Execute the setup script
chmod 755 *.sh # not sure why these were not already executable sudo bash ./setup_xmc1151_dvt.sh
- Select the proper option: 1)Install
Auto-load MIC-3753 Driver
- symlink the drivers into the kernel directory
sudo ln -s /opt/lvisf/src/MIC3753/linux_driver_source_3.2.7.0_64bit/drivers/bin/*.ko /lib/modules/2.6.33.9-rt31.79.el6rt.x86_64/kernel/drivers/misc/
- add to our startup script /etc/sysconfig/modules/lvisf.modules
- modify the file
sudo nano /etc/sysconfig/modules/lvisf.modules
- add the following
# start the MIC-3753 drivers /sbin/modprobe biokernbase /sbin/modprobe bio1753
Auto-load Plx9030 Driver
NOTE: (in lieu of step 2 below) I had this not work when I manually edited it the first time. A copy of the file is in the repository, you could just copy that file in:
sudo cp /opt/lvisf/src/lvisF/scripts/lvisf.modules /etc/sysconfig/modules/lvisf.modules
- symlink the driver into the kernel so it will come up on boot
sudo ln -s /opt/lvisf/src/PLX/PlxSdk/Driver/Source.Plx9000/Output/Plx9030.ko /lib/modules/2.6.33.9-rt31.79.el6rt.x86_64/kernel/drivers/misc/
- add to our startup script /etc/sysconfig/modules/lvisf.modules
- modify the file
sudo nano /etc/sysconfig/modules/lvisf.modules
- add the following
# start the Plx9030 syncclock32 driver /sbin/modprobe Plx9030
- symbolically link in our rc.local into /etc/rc.local
- remove the original
# remove the symlink sudo rm /etc/rc.local # move the original to something else sudo mv /etc/rc.d/rc.local /etc/rc.d/rc.local_orig
- link in the lvisf version
sudo ln -s /opt/lvisf/src/lvisF/scripts/rc.local /etc/rc.d/rc.local sudo ln -s /etc/rc.d/rc.local /etc/rc.local
Installing Boost Library
The boost C++ library is required for the data system. The tarball of the 1.6.2 version is in the repository. Or download the latest from http://www.boost.org/users/download/. Here is the latest as of 2018.01.04 https://dl.bintray.com/boostorg/release/1.66.0/source/boost_1_66_0.tar.bz2
- Uncompress the tar into /opt
cd /opt sudo tar -jxvf /opt/lvisf/src/tars/boost_1_62_0.tar.bz2
- Symbolically link to a generic boost directory
sudo ln -s /opt/boost_1_62_0 /opt/boost
- Chown it all to lvis user
sudo chown -R lvis:lvis /opt/boost_1_62_0
- Set the environment variable for the BOOST_ROOT variable
- Edit the file /home/lvis/.bash_profile
nano /home/lvis/.bash_profile
- Add the line
# Add the boost root directory export BOOST_ROOT=/opt/boost
- Run our profile to load this variable
source /home/lvis/.bash_profile
- Compile the boost libraries
cd /opt/boost ./bootstrap.sh ./b2
Installing USB Temperature Library
Follow the instructions below to compile the USB temperature monitoring device libraries required.
Clean Boot Operations
Load Drivers
- Load the MIC-3753 DIO Drivers
sudo insmod /home/lvis/src/MIC3753/linux_driver_source_3.2.7.0_64bit/drivers/bin/biokernbase.ko sudo insmod /home/lvis/src/MIC3753/linux_driver_source_3.2.7.0_64bit/drivers/bin/bio1753.ko
- Load the SyncClock32 Timer Drivers
cd /home/lvis/src/PLX/PlxSdk/Bin sudo ./Plx_load 9030
Compile Code
- Enter the build directory for lvisf
cd /home/lvis/src/lvisF/build
- Use cmake
cmake ../
- Build code
make
Run Tests
User Space Latency Test
- Enter the testing program
cd /home/lvis/src/lvisF/build/bin/tests/syncclock
- Run the latency program
sudo ./galvMove
Latency Tweaking
- Set CPU affinity to CPU 3 with something like, made it much more consistent since I guess most things go for CPU 0 by default
CPU_SET(3, &cpus);
- Change the scheduler policy, this information is displayed in the syncclock test program
sudo chrt --fifo 99 ./galvMove
- Isolate a CPU, and then have the code attach itself to that cpu
- In /etc/grub.conf add this command to the kernel command line, so all user space code will avoid CPU 2 and 3 in this example, leaving 0 and 1 for the OS
isolcpus=2,3
- Change the program such that
CPU_SET(3, &cpus);
- Run the program with chrt or not
sudo chrt --fifo 99 ./galvMove
- Set the timer card interrupt affinity to a single CPU (not the prime)
sudo bash -c 'echo 2 > /proc/irq/19/smp_affinity'
- Using 2 here put the interrupt 19 servicing on CPU 1 exclusively
- More low latency items to add to the kernel command line for /etc/grub.conf
rhgb nousb quiet isolcpus=2,3 nosoftlockup mce=ignore_ce audit=0
- nosoftlockup disables backtrace functionality
- mce=ignore_ce ignores corrected errors
- audit=0 disables kernel components of the audit subsystem
Testing
- Testing delays
- Install perf
sudo yum install perf
- Run this command to record some latency data
cd /opt/lvisf/src/lvisF/build/bin/tests/syncclock sudo perf sched record -o /dev/shm/perf.data chrt --fifo 99 ./galvMove
- Run the following to read the data
sudo perf sched -i /dev/shm/perf.data latency > /tmp/perf.latency.report
Show Interrupt Status
- This command shows interrupts for each core / CPU
cat /proc/interrupts # notice that CPU1 has all the timer card interrupts (PLx9030) [lvis@lvisf syncclock]$ cat /proc/interrupts CPU0 CPU1 CPU2 CPU3 0: 125 0 0 0 IR-IO-APIC-edge timer 8: 1 0 0 0 IR-IO-APIC-edge rtc0 9: 0 0 0 0 IR-IO-APIC-fasteoi acpi 18: 1968 0 0 0 IR-IO-APIC-fasteoi eth2 19: 0 2021327 0 0 IR-IO-APIC-fasteoi Plx9030 24: 0 0 0 0 DMAR_MSI-edge dmar0 25: 0 0 0 0 DMAR_MSI-edge dmar1 26: 2 0 0 0 IR-HPET_MSI-edge hpet2 27: 0 0 0 0 IR-HPET_MSI-edge hpet3 28: 0 0 0 0 IR-HPET_MSI-edge hpet4 29: 0 0 0 0 IR-HPET_MSI-edge hpet5 34: 235 0 0 0 IR-PCI-MSI-edge i915 35: 0 0 0 0 IR-PCI-MSI-edge ahci 36: 7971 0 0 0 IR-PCI-MSI-edge ahci NMI: 0 0 0 0 Non-maskable interrupts LOC: 20569 29227 10924 50493 Local timer interrupts SPU: 0 0 0 0 Spurious interrupts PMI: 0 0 0 0 Performance monitoring interrupts IWI: 0 0 0 0 IRQ work interrupts RES: 324 194 372 353 Rescheduling interrupts CAL: 47 197 178 211 Function call interrupts TLB: 142 610 312 1 TLB shootdowns TRM: 0 0 0 0 Thermal event interrupts THR: 0 0 0 0 Threshold APIC interrupts MCE: 0 0 0 0 Machine check exceptions MCP: 1 1 1 1 Machine check polls ERR: 0 MIS: 0
Test Results
- 2016.04.27: Ran a test for 2.7 hours. We missed 46 shots out of 49097902 at 5khz (200 useconds) so as far as actually reacting to the interrupt, we are in good shape.
- Time to make a build 1 data system and get better numbers as we do all our tasks.