david.rabine

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.
====== Bash Style Archive ======

===== Field Script =====

==== Prerequisites ====

<code>
sudo apt-get install hfsprogs

## NOTE is saw that we should disable journaling on the external drive before writing with Linux

sudo mount -t hfsplus -o force,rw /dev/sdXY /media/mntpoint
# or
sudo mount -t hfsplus -o remount,force,rw /mount/point
</code>


==== HowTo ====

  - Attach flight drive to archive CPU
  - Attach mirror drive to rsync to (either HFS+ journaling disabled or EXT4)
  - Attach archive drive which will get the compressed mjd tar file for each day
  - rsync the flight data to the mirror drive (directory by directory at top level)
  - run compress script (see below) to generate a MMMMM_0.tar.gz for each flight day on the archive drive

later you can add MMMMM_[1-z].tar.gz for any additional files that were missed?

==== Database ====

Ideas for this in the field

  - Can we generate a csv (white space or commas?) data base of all the files that are on original and mirror drives?
    - Disk Label, Disk Serial Number would be additional fields to the md5deep with size and access time information so we can quickly assess how much of each kind of data we have (oh top level directory type stats also:  gps, imu, camera, rtlvis running stats for all the dirs one down from Mission level

How about
  - Just have the file name either ARCHIVE_DRIVESERIALNUMBER.md5.txt for each drive that is an archive (have blown up the md5deep information within each tar?)
  - for the orignal and mirror do MIRROR_DRIVESERIALNUMBER.md5.txt with the information (append on any new files not already in the file... touch the file if non-exists)


Ingest script:
  - Check the mount point to see if it is FLIGHT, MIRROR or ARCHIVE (want to expand this to have GPS, APPLANIX also?  Could also have specific drive serial numbers that are known data drives (USB sticks or Applanix CF cards for instance))
  - Grab the drive Make and Serial Number to use in file name creation
  - Ingest Logic on each file
    - Does the meta data exist? MIRROR_DRIVESERIALNUMBER.meta.txt  If not.. just touch it!
    - How to avoid double md5deep calculation ?  (if you do the -X it calculates and compares, then we could just run on the output of that... maybe that's a special --switch to make it go faster)
      - Do it and then see if entry is already in the file... if so skip, if not add  This will catch files with different times, etc...  I think this makes the most sense, least likely to have an issue and defaults to ingest this data.


===== Compress All Dirs To A Folder =====

<code bash>
lvis@lvis-dell-xps:~/bin$ more compress_all_mjd.sh 
#!/bin/bash

echo "Usage $0 <input dir> <output dir>"

M_INPUTDIR=$(echo $1 | sed '/\/$/! s|$|/|')  # add a slash if necessary
M_OUTPUTDIR=$(echo $2 | sed '/\/$/! s|$|/|') # found online

echo Input: $M_INPUTDIR
echo Output: $M_OUTPUTDIR

cd $M_INPUTDIR
for f in *; do
    if [ -d "$f" ]; then
        M_BASENAME=$(basename $f)
        re='^[0-9]+$'  # check if this is a just a number (day mjd) (found online)
        if [[ $M_BASENAME =~ $re ]] ; then
           M_TARFILE="${M_OUTPUTDIR}${M_BASENAME=}_raw_data.tar.gz"
           echo "Run on $f which is $M_BASENAME"
           time tar cf - $f | pigz > $M_TARFILE
        fi
    fi
done

</code>

===== Mount Tar As Filesystem =====

  * [[https://csatlas.com/mount-tar-file-ubuntu/]] <- **archivemount** HOWTO

Options we might be interested in
<code>
       -o readonly
              disable write support

       -o nobackup
              remove archive file backups
</code>

  - Install if not already on the system <code>
sudo apt install -y archivemount
</code>
  - Create a mount point <code>
mkdir /tmp/myfiles
</code>
  - Mount the tar file as read only <code>
# read / write mount which makes a .orig file
archivemount myfiles.tar.gz /tmp/myfiles

# read / write mount without any backup
archivemount -o nobackup myfiles.tar.gz /tmp/myfiles

# read only
archivemount -o readonly myfiles.tar.gz /tmp/myfiles

</code>
  - //Read data out of archive as if it were a file//
  - Remove the mount <code>
cd && umount /tmp/myfiles
</code>