david.rabine

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.
====== RTP Airborne Server Build ======

  * FIXME Need to start with ubuntu server, not desktop so the IP address stuff is fixed / static
  * OR figure out how to fix this desktop install

Try this as your netplan file:
<code>
# Let NetworkManager manage all devices on this system
network:
  version: 2
  renderer: NetworkManager
  ethernets:
    eno1:
      renderer: networkd
      match:
        name: eno1
      addresses: [10.4.1.10/24]
      gateway4: 10.4.1.254
      nameservers:
        search: [nasa.gov]
        addresses: [128.183.10.134, 128.183.50.17, 8.8.8.8]
</code>

===== Base Server Install =====

  * Probably need more swap space than 2GB.  Have 64GB of ram so probably should have more like 32GB of swap at least.  The multi-threaded zoom 12 tile generation ran into ram / swap issues before finishing with only 2GB available.  Wonder if you can add temp swap space?

Could try this:  [[https://linuxize.com/post/how-to-add-swap-space-on-ubuntu-18-04/]]

==== Expand Swap Space to 32GB ====

  - Turn off swap so we can resize the swapfile <code>
sudo swapoff -a
</code>
  - Create our 32GB swapfile <code>
sudo fallocate -l 32G /swapfile
</code>
  - Offically make this a swap file <code>
sudo mkswap /swapfile
</code>
  - Check **/etc/fstab** that swap is in there (it is)
  - Can
    - Turn on manually with <code>
sudo swapon /swapfile
</code>
    - OR just reboot and it should show up as activated

===== First Things =====

  - Update <code>
sudo apt update
</code>
  - Upgrade <code>
sudo apt upgrade
</code>

===== ssh Server =====

<code>
sudo apt-get install openssh-server
sudo systemctl enable ssh
sudo systemctl start ssh
</code>

===== KDE Display =====

I am still a big fan of this display

<code>
sudo apt install kde-full
sudo apt install kdevelop
</code>
===== Basics =====


  - Start with a base version of Ubuntu 20.04 LTS
  - Install base packages <code>
sudo apt -y install htop jed screen
sudo apt -y install gimp
sudo apt -y install net-tools
sudo apt-get -y install build-essential
sudo apt-get -y install autoconf automake cmake gdb git libffi-dev zlib1g-dev libssl-dev
sudo apt -y install ntpdate
sudo apt -y install unzip pigz md5deep
sudo apt -y install gfortran
sudo apt-get -y install libhdf5-serial-dev
sudo apt -y install hdf5-tools
sudo apt-get -y install libopenblas-dev
sudo apt-get -y install libnetcdf-dev libnetcdff-dev
sudo apt -y install libtiff-dev
</code>
  - **EXFAT** read and write <code>
sudo add-apt-repository universe
sudo apt update
sudo apt -y install exfat-fuse exfat-utils
</code>
  - Install **python2** and requirements for Sarah's python code <code>
sudo apt -y install python2
</code>
    - Install **python3** modules using **pip**<code>
sudo apt -y install python3-pip
pip3 install numpy --user
pip3 install h5py --user
pip3 install scipy --user
pip3 install matplotlib --user
</code>
  - KML browser will want this <code>
sudo apt -y install firefox
sudo apt -y install imagemagick
</code>
  - Install **GIS** code <code>
sudo add-apt-repository -y ppa:ubuntugis/ppa
sudo apt update
sudo apt -y install gdal-bin  # I did also do this from source, but not necessary
# sudo apt install gmt # build from source did not work ubuntu 20.04
sudo apt -y install qgis
</code>
  - Google Earth
    - Grab the installer <code>
mkdir ~/down
cd ~/down
wget -O ~/down/google-earth.deb https://dl.google.com/dl/earth/client/current/google-earth-pro-stable_current_amd64.deb
</code>
    - Install <code>
sudo dpkg -i ~/down/google-earth.deb
</code>
===== Install Source Code =====

===== Build dlrdem From Source =====

NOTE:  Jason is an excellent programmer and his code rocks! 8-)

  - Expand source <code>
mkdir ~/repos
cd ~/repos
tar -zxvf ~/down/dlrdem_src.tar.gz
</code>
  - Use **cmake** to create the compile configurations <code>
cd ~/repos/dlrdem/
mkdir build
cd build
cmake ..
</code>
  - Compile (you can use -jCORES) <code>
CPUCOUNT=$(grep -c   "^processor" /proc/cpuinfo)
make -j $CPUCOUNT
</code>
  - Link into our **~/bin** dir <code>
mkdir ~/bin
ln -s ~/repos/dlrdem/build/interpolate_dlrdem ~/bin
</code>
  - Copy the DEM folder from another computer <code>
rsync -vaP lvis@LOCALIP:dem/ ~/dem/
</code>
===== Build VEGAS From Source =====

  - Expand source <code>
mkdir ~/repos
cd ~/repos
tar -zxvf ~/down/vegas_source.tar.gz
</code>
  - Use **cmake** to create the compile configurations <code>
cd ~/repos/VEGAS/code-2021-07-14
rm -rf build
mkdir build
cd build
cmake ..
</code>
  - Compile (you can use -jCORES) <code>
CPUCOUNT=$(grep -c   "^processor" /proc/cpuinfo)
make -j $CPUCOUNT
</code>
  - Link into our **~/bin** dir <code>
mkdir ~/bin
ln -s ~/repos/VEGAS/code-2021-07-14/build/vegas ~/bin/
</code>

===== Build RTP Software =====

==== QLV2 Utilities ====

  - Uncompress the source <code>
mkdir ~/repos
cd ~/repos
unzip ~/down/lvisf-master.zip
</code>
  - Compile (I should make this part of the cmake stuff) <code>
cd ~/repos/lvisf-master/rtp/rtp_lvisf2ql2
</code>
  - Compile <code>
make
</code>
  - Link the binaries <code>
ln -s ~/repos/lvisf-master/rtp/rtp_lvisf2ql2/rtp_lvisf2ql2 ~/bin/
ln -s ~/repos/lvisf-master/rtp/rtp_lvisf2ql2/rtp_ql2filter ~/bin/
ln -s ~/repos/lvisf-master/rtp/rtp_lvisf2ql2/rtp_ql2vegas ~/bin/
</code>

==== Real Time Traj Capture ====

  - Go to <code>
/home/lvis/repos/lvisf-master/bin/utils/rtpd_nav
</code>
  - Compile <code>
make
</code>
  - Link daemon <code>
ln -s ~/repos/lvisf-master/bin/utils/rtpd_nav/rtpd_nav ~/bin/
</code>
===== Build GMT From Source =====

**NOTES:**
  * I attempted to get **OpenMP** working on **GMT** but it turns out that it doesn't even show up in the **nearneighbor** module so this really doesn't help out.  So multi-thread GMT, not so much, but what I ended up doing is just launching lots of threads, and that works fine.
  * Ubuntu **gmt** was not working so compiling from source is mandatory to get valid results (my range on the data sets made no sense, so some kind of default variable size issue probably)

  - Install some more packages required to compile <code>
sudo apt -y install subversion build-essential cmake libcurl4-gnutls-dev libnetcdf-dev libgdal-dev libfftw3-dev libpcre3-dev liblapack-dev libblas-dev
sudo apt -y install libglib2.0-dev ffmpeg graphicsmagick
</code>
  - Go to **src** directory <code>
mkdir ~/src
cd ~/src
</code>
  - Clone the **git** repository <code>
git clone https://github.com/GenericMappingTools/gmt.git
</code>
  - Run **cmake** <code>
cd gmt
mkdir build
cd build
cmake -D GMT_USE_THREADS=true -D GMT_ENABLE_OPENMP=true ..
</code>
  - Build <code>
CPUCOUNT=$(grep -c   "^processor" /proc/cpuinfo)
make -j $CPUCOUNT
</code>
  - Install <code>
sudo make install
</code>

===== Build LVIS Code =====

==== rawlvis2quicklook ====

  - Get the boost development libraries <code>
sudo apt-get install libboost-all-dev
</code>
  - Compile our raw lvis to quicklook program
    - Get into directory <code>
cd /home/lvis/repos/lvisf-master/bin/utils/rawlvis2quicklook
</code>
    - Make symbolic links <code>
ln -s ../../../lib/lvisf_utils/lvisf_utils.* .
ln -s ../../../bin/utils/parseData/xmcParseMeta.cpp
</code>
    - Compile this code together <code>
g++ -Wall -O2 \
    lvisf_utils.cpp \
    rawlvis2quicklook.cpp \
    xmcParseMeta.cpp \
   -I../../../include \
   -I../../../bin/utils/parseData \
   -o rawlvis2quicklook
</code>
  - Link in the new binary <code>
ln -s ~/repos/lvisf-master/bin/utils/rawlvis2quicklook/rawlvis2quicklook ~/bin/
</code>

====== NFS Link Backend to RTP Server ======

===== Setup the Backend =====

  * From:  [[https://www.digitalocean.com/community/tutorials/how-to-set-up-an-nfs-mount-on-centos-6]]

  - Install these packages on the Centos 6 back end by hand (this was a little painful) since end of life for CentOS 6.5 <code>
sudo su  # to get sudo prompt
rpm -ivh libgssglue-0.1-11.el6.x86_64.rpm 
rpm -ivh libtirpc-0.2.1-6.el6_4.x86_64.rpm 
rpm -ivh rpcbind-0.2.0-11.el6.x86_64.rpm
rpm -ivh keyutils-1.4-5.el6.x86_64.rpm
rpm -ivh libevent-1.4.13-4.el6.x86_64.rpm
rpm -ivh --nodeps  nfs-utils-lib-1.1.5-6.el6.x86_64.rpm 
rpm -ivh nfs-utils-1.2.3-39.el6.x86_64.rpm 
</code>
  - Export our internal drive with an **/etc/exports** file that looks like this <code>
/mnt/internal               10.4.1.11(ro,sync,all_squash,no_subtree_check)
</code>
  - Enable the NFS service to be launched on startup (Turn On a Service for the Default Run Levels (2,3,4,5) [[https://www.liquidweb.com/kb/chkconfig-command-examples-for-red-hat-and-centos/|from here]]) <code>
chkconfig nfs on 
</code>
  - Kick off the NFS service (NFS is dependent on RPC) <code>
service rpcbind start
service nfs start
</code>
  - Show what is being shared up with <code>
showmount -e 10.4.1.2
</code>

===== Setup the RTP NFS Client =====

  * From:  [[https://www.digitalocean.com/community/tutorials/how-to-set-up-an-nfs-mount-on-ubuntu-20-04]]

  - Install these <code>
sudo apt update
sudo apt install nfs-common
</code>
  - Make a mount point on our server <code>
sudo mkdir -p /nfs/lvisf/internal
</code>
  - Mount the directory
    - Add this line to the **/etc/fstab** to mount this on startup <code>
</code>
    - **OR** Mount the remote directory manually from the command line<code>
sudo mount 10.4.1.2:/mnt/internal /nfs/lvisf/internal
</code>

====== HTTP Server ======

Source:  [[https://www.digitalocean.com/community/tutorials/how-to-install-the-apache-web-server-on-ubuntu-18-04-quickstart]]

  - Install httpd <code>
sudo apt -y update
sudo apt -y install apache2
</code>
  - Adjust firewall if necessary (was not necessary (see link above) but this shows apache already through the firewall <code>
sudo ufw app list
</code>
    - See apache is on the list <code>
Available applications:
  Apache
  Apache Full
  Apache Secure
  CUPS
  OpenSSH
</code>
  - Create a directory where the output can be written to <code>
sudo chmod 775 /var/www/html
sudo chgrp -R lvis /var/www/html
ln -s /var/www/html
mkdir /var/www/html/MJD
</code>
  - Just need to add this line to each html overlay desired
    - Leaflet is like this <code>
       // Overlay layers (TMS)
        var lyr = L.tileLayer('http://10.4.1.11/59431/{z}/{x}/{y}.png', {tms: 1, opacity: 0.7, attribution: "", minZoom: 0, maxZoom: 10});
</code>
    - Openlayers is like this <code>
url: 'http://10.4.1.11/59431/{z}/{x}/{-y}.png'
</code>

====== OSM Tile Server Setup ======

  * [[https://switch2osm.org/serving-tiles/manually-building-a-tile-server-20-04-lts/]] will show you how to setup a local tile server to create tiles in real time based on the OSB planet data base file you are now downloading (PBF snapshot of the Earth file)  I wish they had the topo one?
  * [[https://www.postgresql.org/docs/6.3/c0702.htm]] alternate location of the data base, as it will not fit on the root drive so refer to this link
  * Download entire earth PBF file here: [[https://planet.openstreetmap.org/]]

===== Requirements =====

  - Create the user account to render the tiles
    - Open a sudo shell <code>
sudo -i
</code>
    - Add the user and set permissions<code>
adduser renderaccount
# answer the questions and then
usermod -aG sudo renderaccount
exit
</code>
  - Log in to this account
  - Install some required software <code>
sudo apt -y install libboost-all-dev git tar unzip wget bzip2 build-essential autoconf libtool libxml2-dev libgeos-dev libgeos++-dev libpq-dev libbz2-dev libproj-dev munin-node munin protobuf-c-compiler libfreetype6-dev libtiff5-dev libicu-dev libgdal-dev libcairo2-dev libcairomm-1.0-dev apache2 apache2-dev libagg-dev liblua5.2-dev ttf-unifont lua5.1 liblua5.1-0-dev
</code>

===== Installing postgresql / postgis =====

  - Install data base packages needed <code>
sudo apt -y install postgresql postgresql-contrib postgis postgresql-12-postgis-3 postgresql-12-postgis-3-scripts
</code>
  - Migrate our database to the other drive as it has way more space
    - Check WHERE the current data base is located
      - Fire up **psql** <code>
sudo -u postgres psql
</code>
      - Show the directory being used <code>
SHOW data_directory;
</code>
      - Result for us today is <code>
/var/lib/postgresql/12/main
</code>
    - Shut down the **postgresql** server so we can modify the data base location <code>
sudo systemctl stop postgresql
sudo systemctl status postgresql
</code>
    - Copy the existing data base to the other drive <code>
sudo rsync -av /var/lib/postgresql /mnt/rtp
</code>
    - Move the original data just in case <code>
sudo mv /var/lib/postgresql/12/main /var/lib/postgresql/12/main.bak
</code>
    - Point the database to the new location
      - Edit the configuration file <code>
sudo nano /etc/postgresql/12/main/postgresql.conf
</code>
      - Change
        - Original <code>
data_directory = '/var/lib/postgresql/12/main'          # use data in another directory
</code>
        - To this <code>
data_directory = '/mnt/rtp/postgresql/12/main'          # use data in another director
</code>
    - Restart the data base <code>
sudo systemctl start postgresql
sudo systemctl status postgresql
</code>
    - Log in and check the data base
      - Log into the server<code>
sudo -u postgres psql
SHOW data_directory;
</code>
      - Results show that we have moved the database! <code>
       data_directory        
-----------------------------
 /mnt/rtp/postgresql/12/main
(1 row)
</code>
      - Quit out <code>
CONTROL-D or \q to quit
</code>
  - Create our database
    - Create the actual database<code>
sudo -u postgres -i
createuser renderaccount # answer yes for superuser (although this isn't strictly necessary)
createdb -E UTF8 -O renderaccount gis
</code>
    - A bunch of database commands to alter the database <code>
psql
\c gis
CREATE EXTENSION postgis;
CREATE EXTENSION hstore;
ALTER TABLE geometry_columns OWNER TO renderaccount;
ALTER TABLE spatial_ref_sys OWNER TO renderaccount;
\q
exit
</code>

===== Installing osm2pgsql =====

<code>
sudo apt -y install osm2pgsql
</code>

===== Installing Mapnik =====

<code>
sudo apt -y install autoconf apache2-dev libtool libxml2-dev libbz2-dev libgeos-dev libgeos++-dev libproj-dev gdal-bin libmapnik-dev mapnik-utils python3-mapnik python3-psycopg2 python3-yaml
</code>

===== Install mod_tile and renderd =====

  - Install from code <code>
mkdir ~/src
cd ~/src
git clone -b switch2osm git://github.com/SomeoneElseOSM/mod_tile.git
cd mod_tile
./autogen.sh
</code>
  - Make and install <code>
./configure
CPUCOUNT=$(grep -c   "^processor" /proc/cpuinfo)
make -j $CPUCOUNT
sudo make install
sudo make install-mod_tile
sudo ldconfig
</code>

===== Stylesheet configuration =====

  - Grab stuff <code>
cd ~/src
git clone git://github.com/gravitystorm/openstreetmap-carto.git
cd openstreetmap-carto
</code>
  - Add some more things <code>
sudo apt -y install npm
sudo npm install -g carto
carto -v
</code>
  - Response should be at least <code>
1.2.0
</code>
  - Use this to convert something Mapnik can understand <code>
carto project.mml > mapnik.xml
</code>

===== Loading data =====

==== Test Data ====

NOTE:  This does give you water and land at least the basics, this is probably good enough.

  - Grab some sample data to get a taste and see if we can get this working <code>
mkdir ~/data
cd ~/data
wget https://download.geofabrik.de/asia/azerbaijan-latest.osm.pbf
</code>
  - Import the data and use all the threads we have <code>
osm2pgsql -d gis --create --slim  -G --hstore --tag-transform-script ~/src/openstreetmap-carto/openstreetmap-carto.lua -C 2500 --number-processes 16 -S ~/src/openstreetmap-carto/openstreetmap-carto.style ~/data/azerbaijan-latest.osm.pbf
</code>

==== Entire Earth ====

VanGilst said THIS WILL TAKE A WHILE!  You could leave it alone, it looks pretty good as is.  But, let's break it.

<code>
time osm2pgsql -d gis --create --slim  -G --hstore --tag-transform-script ~/src/openstreetmap-carto/openstreetmap-carto.lua -C 2500 --number-processes 16 -S ~/src/openstreetmap-carto/openstreetmap-carto.style /mnt/rtp/tmp/planet-220117.osm.pbf
</code>

==== Subdivisions ====

If you want a smaller **pbf** file which should take less time to ingest, you can grab the subdivisions here:
  * [[https://download.geofabrik.de/]]  <- Get smaller PBF files here! :!:

<code>
time osm2pgsql -d gis --create --slim  -G --hstore --tag-transform-script ~/src/openstreetmap-carto/openstreetmap-carto.lua -C 2500 --number-processes 16 -S ~/src/openstreetmap-carto/openstreetmap-carto.style ~/data/greenland-latest.osm.pbf
time osm2pgsql -d gis --append --slim  -G --hstore --tag-transform-script ~/src/openstreetmap-carto/openstreetmap-carto.lua -C 2500 --number-processes 16 -S ~/src/openstreetmap-carto/openstreetmap-carto.style ~/data/antarctica-latest.osm.pbf
</code>


===== Creating indexes =====

  - Run this <code>
cd ~/src/openstreetmap-carto/
psql -d gis -f indexes.sql
</code>
  - And you will get back "CREATE INDEX" 14 times

===== Shapefile download =====

This gets the basic water and land information

<code>
cd ~/src/openstreetmap-carto/
scripts/get-external-data.py
</code>

===== Fonts =====

<code>
sudo apt -y install fonts-noto-cjk fonts-noto-hinted fonts-noto-unhinted ttf-unifont
</code>

===== Setting up your webserver =====

==== renderd ====

  - Setup renderd configuration <code>
sudo nano /usr/local/etc/renderd.conf
</code>
    - NOTE: Since we used **renderaccount** and have plenty of memory, all the defaults are OK
  - This line, verify this is correct by doing an **ls** <code>
XML=/home/renderaccount/src/openstreetmap-carto/mapnik.xml
</code>
  - Change the version from 3.0 to 3.1 as of this writing
    - This <code>
plugins_dir=/usr/lib/mapnik/3.0/input
</code>
    - Should be this <code>
plugins_dir=/usr/lib/mapnik/3.1/input
</code>


==== apache2 ====

  - Some housekeeping <code>
sudo mkdir /var/lib/mod_tile
sudo chown renderaccount /var/lib/mod_tile

sudo mkdir /var/run/renderd
sudo chown renderaccount /var/run/renderd
</code>
  - Edit this file <code>
sudo nano /etc/apache2/conf-available/mod_tile.conf
</code>
    - And add the following line and then exit out of the editor<code>
LoadModule tile_module /usr/lib/apache2/modules/mod_tile.so
</code>
  - Run this to enable the tile mod <code>
sudo a2enconf mod_tile
</code>
  - Edit this file and add these lines where they go <code>
sudo nano /etc/apache2/sites-available/000-default.conf
</code>
    - And add the following between the “ServerAdmin” and “DocumentRoot” lines: <code>
LoadTileConfigFile /usr/local/etc/renderd.conf
ModTileRenderdSocketName /var/run/renderd/renderd.sock
# Timeout before giving up for a tile to be rendered
ModTileRequestTimeout 0
# Timeout before giving up for a tile to be rendered that is otherwise missing
ModTileMissingRequestTimeout 30
</code>
  - Reload apache2 twice <code>
sudo service apache2 reload
sudo service apache2 reload
</code>

==== Local test ====

  - Kick off **renderd** in the foreground just running it on a console <code>
sudo mkdir /var/lib/mod_tile
sudo chown renderaccount /var/lib/mod_tile
sudo mkdir /var/run/renderd
sudo chown renderaccount /var/run/renderd
renderd -f -c /usr/local/etc/renderd.conf
</code>
  - Go to your local browser and see if this renders an image:  [[http://localhost/hot/0/0/0.png]]

After finding the 3.0 to 3.1 change in the renderd configuration file, it worked for me!

==== renderd in the background ====

  - Check that **renderaccount** is the RUNASUSER in this file <code>
nano ~/src/mod_tile/debian/renderd.init
</code>
  - Copy this into our init directory <code>
sudo cp ~/src/mod_tile/debian/renderd.init /etc/init.d/renderd
sudo chmod u+x /etc/init.d/renderd
sudo cp ~/src/mod_tile/debian/renderd.service /lib/systemd/system/
</code>
  - Start the service <code>
sudo /etc/init.d/renderd start
</code>
  - Set so that it starts on startup <code>
sudo systemctl enable renderd
</code>

===== Viewing local tiles =====

  * Good info: [[https://switch2osm.github.io/using-tiles/getting-started-with-leaflet/]]
  * Try our locally hosted map now!  [[http://10.4.1.10/rtp_leaflet.html]]

  - Put this into **leaflet.html** in your html folder <code html>
<!DOCTYPE HTML>
<html lang="en">
  <head>
    <meta charset="utf-8" />
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <script src="https://unpkg.com/leaflet@1.7.1/dist/leaflet.js"></script>
    <link rel="stylesheet" href="https://unpkg.com/leaflet@1.7.1/dist/leaflet.css" />
    <style>
      html, body {
        height: 100%;
        padding: 0;
        margin: 0;
      }
      #map {
        /* configure the size of the map */
        width: 100%;
        height: 100%;
      }
    </style>
  </head>
  <body>
    <div id="map"></div>
    <script>
      // initialize Leaflet
      var map = L.map('map').setView({lon: 0, lat: 0}, 2);

      // add the OpenStreetMap tiles
      L.tileLayer('https://{s}.tile.openstreetmap.org/{z}/{x}/{y}.png', {
        maxZoom: 19,
        attribution: '&copy; <a href="https://openstreetmap.org/copyright">OpenStreetMap contributors</a>'
      }).addTo(map);

      // show the scale bar on the lower left corner
      L.control.scale({imperial: true, metric: true}).addTo(map);

      // show a marker on the map
      L.marker({lon: 0, lat: 0}).bindPopup('The center of the world').addTo(map);
    </script>
  </body>
</html>
</code>
  - And modify the line that renders the tile layer with this line <code>
L.tileLayer('http://localhost/hot/{z}/{x}/{y}.png', {
</code>