david.rabine

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.
====== ARCSIX Field Processing ======

  * [[code:work:lvisf:2024:field_processing:arcsix2:summary:start|ARCSIX-2 Data Summary]]
  * [[code:work:lvisf:2024:field_processing:arcsix2:summary:alignment|ARCSIX-2 Alignment Summary]]
  * Backup Drives "/media/lvis/A6_BACKUP01"
  * Backup Drives "/media/lvis/A6_BACKUP02"

FIXME - Make the ingest log how much space is free on the disk each time it syncs

FIXME - TODO:
  * Change the IMU names to 2024
  * Pluck out some low altitude data for Michelle
    * Low level (1500' & 4000')
    * Spiral down reasonable ice<code>
IDL> print,min(c.bctime)
       48708.312
IDL> print,max(c.bctime)
       48878.315
</code>
    * Descent during porpoise with clouds?
    * East vs West area
  * Find photo that matches the data that Michelle generated the image for (lat/lon)
    * Lat 84 42 15 N Lon 65 41 W ( -65.68333333 84.70416667 ) <code>
x=[-65.68333333,-65.68333333] & y = [84.70416667,84.70416667]
; zm in 52690 (14:38:10) there abouts and 53350 (14:49:10)
</code>
    * File is:  lvis_20240528_143800_00000306.dat
  * Add final scripts to make symbolic links for 510i, CAM1, P1
  * Fix this page:  How straight forward is this? (schedule with dinner included)
  * PDF / Print this so you have a hard copy and a local version

  * [[code:work:lvisf:2024:field_processing:arcsix1:summary:start|ARCSIX-1 Data Summary]] ([[code:work:lvisf:2024:field_processing:arcsix:summary:start|ARCSIX-1 Data Summary original page]])

===== Aircraft Data =====

  - Backup lvisf drive
  - Backup camera drive
  - Backup rtp drive except for 'rtlvis' folder as it is a duplicate
  - Create proc folder for the new mjd
    - Create subfolders
    - Copy the ql2 files from aircraft RTP drive into v2 directory
  - Generate tiles
  - 

===== Ground GPS =====

  * [[code:work:lvisf:2024:field_processing:arcsix:javad_base_script|Javad Basestation Script]]

===== Specifics =====

<code>
# required variables
export mjd=60461
export lvisf_path=$(find /mnt/archive/lvisf/ -type d -name "${mjd}")
ln -s $lvisf_path /mnt/archive/mjd/
</code>


==== Copy ====

=== Copy lvisf External Drive ===

Ingest bash script:  {{ :code:work:lvisf:2024:field_processing:arcsix:ingest_lvisf_external.zip |}}

  - Attach drive to computer (do not let it mount)
  - Run the following script <code>
ingest_lvisf_external
</code>

=== Copy 510 card ===

<code>
ingest_lvisf_card
</code>

=== Copy 610 usb ===

<code>
ingest_lvisf_usb
</code>

=== Copy Camera drive ===

<code>
ingest_lvisf_camera
</code>

=== Copy RTP drive ===

<code>
ingest_lvisf_rtp
</code>

=== Create Symbolic Links ===

<code>
create_archive_links.sh ${mjd}
</code>


=== GPS Ground Data ===

  - Laptop out to shack
  - run <code>
javad_get_bases
</code>
  - Plug ethernet into switch
  - Copy data from laptop via rsync to raid (on laptop) (open cygwin shell) <code>
rsync -vaP ~/gndnav/ lvis@192.168.0.48:/mnt/archive/gndnav/
</code>
  - Sync archive to external GROUNDNAV ssd <code>
rsync -vaP /mnt/archive/gndnav/ /media/lvis/GROUND_NAV/gndnav/
</code>

==== Care Package Creation ====

=== MJD Skeleton ===

<code>
create_mjd_care_package.sh ${mjd}
</code>

==== Quality ====

=== Generate KMZs ===

== On Ground If RTP Fails ==

  - Create directory structure and symbolically link RTP ql2 files from aircraft <code>
# MOUNT proc drive
sudo mount /dev/disk/by-label/LVISF_PROC /mnt/proc
mkdir -p /mnt/proc/lvisf/${mjd}/v2
cd /mnt/proc/lvisf/${mjd}
ln -s /mnt/archive/rtp/mjd/${mjd}/v2/*.ql2 /mnt/proc/lvisf/${mjd}/v2 
</code>
  - Kick off processing <code>
time ~/field_tools/local/rtp_local_proc.sh ~/field_tools/local/config_610i_arcsix2024.cfg ${mjd}
</code>
  - Channel mixed processing <code>
time ~/field_tools/local/rtp_local_proc.sh ~/field_tools/local/config_610i_arcsix2024_chanmix.cfg ${mjd}
</code>

If you need to start over
<code>
cd /mnt/proc/lvisf/${mjd}
~/field_tools/local/clean.sh all
</code>

If you need to use a different trajectory
<code>
cd /mnt/proc/lvisf/${mjd}/traj/
find /mnt/archive/usb/APP610I-2/ -type f -newermt 2024-06-06 -exec cat {} \; > /mnt/proc/lvisf/${mjd}/traj/traj${mjd}_610i.bin
</code>

If you need to run the kmz generation manually
<code>
inst=LVISF
tiff2kml.py height "$inst""$mjd"_Field_Height.kmz "$inst""$mjd"_Field_Height height.png ${HOME}/field_tools/local/LVIS.png coords.txt
rm files/*
tiff2kml.py elevation "$inst""$mjd"_Field_Elevation.kmz "$inst""$mjd"_Field_Elevation elevation.png ${HOME}/field_tools/local/LVIS.png coords.txt
rm files/*
tiff2kml.py elevation_block "$inst""$mjd"_Elevation_5m.kmz "$inst""$mjd"_Elevation_5m ${HOME}/field_tools/local/elev_block2.png ${HOME}/field_tools/local/LVIS.png coords.txt
rm files/*
tiff2kml.py energy "$inst""$mjd"_Field_Energy.kmz "$inst""$mjd"_Field_Energy energy.png ${HOME}/field_tools/local/LVIS.png coords.txt
</code>

== Copy RTP Results ==

This is the standard, only do the above if the onboard RTP generation didn't produce results

  - Create a directory in the **proc** drive <code>
mkdir -p /mnt/proc/lvisf/${mjd}
</code>
  - Copy the files from the RTP archive (which is a copy from the plane) (excluding rtlivs just in case it was somehow copied over) <code>
rsync -vaP --exclude='rtlvis' /mnt/archive/rtp/mjd/${mjd}/ /mnt/proc/lvisf/${mjd}/
</code>
  - Copy all **kmz** files to external drive (optional, but a good idea assuming GROUND_NAV is mounted!) (max depth was because there is an older folder in one of the days to avoid copying those) <code>
find /mnt/proc/lvisf/ -maxdepth 2 -name "*.kmz" -type f -exec cp -a {} /media/lvis/GROUND_NAV/kmz/ \;
</code>

=== Summary Plots ===

  - Scrape the ql2 files <code>
mkdir -p /mnt/proc/summaryplots/${mjd}
scrape_ql2 /mnt/proc/lvisf/${mjd}/v2 /mnt/proc/summaryplots/${mjd}/${mjd} -s 17
</code>
  - Generate the plots <code>
cd /mnt/proc/summaryplots/${mjd}/
~/field_tools/rtp_summaryplots/runall.sh ${mjd}
</code>
  - Copy to external drive
    - Mount GROUND_NAV
    - Sync the files to it <code>
rsync -vaP /mnt/proc/summaryplots/ /media/lvis/GROUND_NAV/summaryplots/
</code>
  - Sync back to incoming at GSFC

=== Mission Meta ===

<code>
ln -s /mnt/archive/mjd/60410_60550_ancillary/mission_meta/coverage/arcsix2024.cfg ~/field_tools/mission_meta/
</code>

  - Modify the configuration file so that the last mjd is captured
    - Edit the config <code>
nano /mnt/archive/mjd/60410_60550_ancillary/mission_meta/coverage/arcsix2024.cfg
</code>
  - Generate mission meta data <code>
cd ~/field_tools/mission_meta
time ./make_mission_data_summary.py arcsix2024.cfg
</code>
  - Copy to external drive
    - Mount GROUND_NAV
    - Sync the files to it <code>
rsync -vaP /mnt/archive/mjd/60410_60550_ancillary/mission_meta/coverage/ /media/lvis/GROUND_NAV/coverage/
</code>

Results end up here:  <code>/mnt/archive/mjd/60410_60550_ancillary/mission_meta/coverage </code>

=== External Drive MJD Tar ===

  - Attached external drive and mount with Ubuntu manager
  - Run this script based on MJD <code>
sleep 2h ; create_mjd_external_archive.sh ${mjd} /media/lvis/A6_BACKUP03/
</code>
  - Copy the camera folder to the external drive.

=== Copy Applanix Logs from External Camera Backup ===

<code>
find /media/lvis/A6_BACKUP04 -maxdepth 2 -type f -iname '*_log.txt'  -exec cp "{}" /media/lvis/GROUND_NAV/applanix_logs/ \;

# to make sure you get ALL txt files (Rob and John named them differently)
find /mnt/archive/camera/id/ -maxdepth 3 -type f -iname '*.txt'  -exec cp "{}" /tmp/applanix_logs/ \;
</code>


==== Raw Data Copy to Gaia ====

  - Copy card data <code>
rsync -vazP /mnt/archive/card/ lvis@gaia.lattice.net:incoming/card/
</code>
  - Copy usb data <code>
rsync -vazP /mnt/archive/usb/ lvis@gaia.lattice.net:incoming/usb/
</code>

==== Verification ====

  * Run md5deep on LVIS raw files
  * Run teqc on GPS raw files
  * Run applanix time check on raw IMU files
  * Camera?  FIXME


==== Final Raid Inventory ====

  * Ensure each drive has a meta and inventory entry
  * Copy inventory and meta files to three places **drive_meta/year/campaign/**
    * camera, card, lvisf, nav, rtp, usb
    * Box, Teams and Wiki <code>
# manual creation of inventory
time_date=$(date '+%Y%m%d')
time_time=$(date '+%H%M%S')
inventory_output="${camera_basepath}/inventory/${time_date}_${time_time}_${camera_diskid}_${linkname}_inventory.txt"
meta_output="${camera_basepath}/meta/${linkname}.txt"
mkdir -p ${camera_basepath}/inventory/ ${camera_basepath}/meta/
meta_path="${camera_basepath}/meta/"
drivemeta=$(df -h | grep ${camera_mountpoint})
echo "${time_date} ${time_time} ${camera_diskid} ${drivemeta}"
echo "${time_date} ${time_time} ${camera_diskid} ${drivemeta}" >> ${meta_output}
echo "find ${camera_mountpoint} -type f -exec ls -l {} \; >> ${inventory_output}"
find ${camera_mountpoint} -type f -exec ls -l {} \; >> ${inventory_output}
</code>
  * **md5deep** will follow symbolic links, so run this <code>
time sudo find /mnt/archive/ -type f -exec md5deep -z {} \; > ~/md5deep_archive2024_all.txt

# 68 hours for 44TB not bad
# /dev/sda2        44T   43T  970G  98% /mnt/archive
real    4033m5.615s
user    1616m48.209s
sys     429m25.964s

# and make a sorted version of the file
cat ~/md5deep_archive2024_all.txt  | sort -k 3 > ~/md5deep_archive2024_all_sorted.txt
</code>