Differences

This shows you the differences between two versions of the page.

--- code:work:lvisf:2022:fieldarchive:start [2022/04/08 17:10] – david
+++ code:work:lvisf:2022:fieldarchive:start [2022/04/08 17:56] (current) – [Database] david
@@ Line 1: / Line 1: @@
 ====== Bash Style Archive ======
+===== Field Script =====
+==== Prerequisites ====
+<code>
+sudo apt-get install hfsprogs
+## NOTE is saw that we should disable journaling on the external drive before writing with Linux
+sudo mount -t hfsplus -o force,rw /dev/sdXY /media/mntpoint
+# or
+sudo mount -t hfsplus -o remount,force,rw /mount/point
+</code>
+==== HowTo ====
+  - Attach flight drive to archive CPU
+  - Attach mirror drive to rsync to (either HFS+ journaling disabled or EXT4)
+  - Attach archive drive which will get the compressed mjd tar file for each day
+  - rsync the flight data to the mirror drive (directory by directory at top level)
+  - run compress script (see below) to generate a MMMMM_0.tar.gz for each flight day on the archive drive
+later you can add MMMMM_[1-z].tar.gz for any additional files that were missed?
+==== Database ====
+Ideas for this in the field
+  - Can we generate a csv (white space or commas?) data base of all the files that are on original and mirror drives?
+    - Disk Label, Disk Serial Number would be additional fields to the md5deep with size and access time information so we can quickly assess how much of each kind of data we have (oh top level directory type stats also:  gps, imu, camera, rtlvis running stats for all the dirs one down from Mission level
+How about
+  - Just have the file name either ARCHIVE_DRIVESERIALNUMBER.md5.txt for each drive that is an archive (have blown up the md5deep information within each tar?)
+  - for the orignal and mirror do MIRROR_DRIVESERIALNUMBER.md5.txt with the information (append on any new files not already in the file... touch the file if non-exists)
+Ingest script:
+  - Check the mount point to see if it is FLIGHT, MIRROR or ARCHIVE (want to expand this to have GPS, APPLANIX also?  Could also have specific drive serial numbers that are known data drives (USB sticks or Applanix CF cards for instance))
+  - Grab the drive Make and Serial Number to use in file name creation
+  - Ingest Logic on each file
+    - Does the meta data exist? MIRROR_DRIVESERIALNUMBER.meta.txt  If not.. just touch it!
+    - How to avoid double md5deep calculation ?  (if you do the -X it calculates and compares, then we could just run on the output of that... maybe that's a special --switch to make it go faster)
+      - Do it and then see if entry is already in the file... if so skip, if not add  This will catch files with different times, etc...  I think this makes the most sense, least likely to have an issue and defaults to ingest this data.
 ===== Compress All Dirs To A Folder =====
@@ Line 29: / Line 75: @@
 </code>
+===== Mount Tar As Filesystem =====
+  * [[https://csatlas.com/mount-tar-file-ubuntu/]] <- **archivemount** HOWTO
+Options we might be interested in
+<code>
+       -o readonly
+              disable write support
+       -o nobackup
+              remove archive file backups
+</code>
+  - Install if not already on the system <code>
+sudo apt install -y archivemount
+</code>
+  - Create a mount point <code>
+mkdir /tmp/myfiles
+</code>
+  - Mount the tar file as read only <code>
+# read / write mount which makes a .orig file
+archivemount myfiles.tar.gz /tmp/myfiles
+# read / write mount without any backup
+archivemount -o nobackup myfiles.tar.gz /tmp/myfiles
+# read only
+archivemount -o readonly myfiles.tar.gz /tmp/myfiles
+</code>
+  - //Read data out of archive as if it were a file//
+  - Remove the mount <code>
+cd && umount /tmp/myfiles
+</code>