Console Programs¶

The primary application of this library is to provide these console scripts to the user to easily execute HiRISE processing procedures.

hiproc¶

Start from .img files and run them through a complete processing chain.

usage: hiproc [-h] [-v] [--logfile LOGFILE] [-k] [--version] [-c] [-p] [-j]
              [--max_workers MAX_WORKERS] [--conf_dir CONF_DIR]
              some.img-file [some.img-file ...]

some.img-file¶: More than one can be listed here.

-h, --help¶: show this help message and exit

-v, --verbose¶: Displays additional information as processing progresses.

--logfile <logfile>¶: The log file to write log messages to in addition to the terminal.

-k, --keep¶: Normally, the program will clean up any intermediary files, but if this option is given, it won’t.

--version¶: Show hiproc version number.

-c, --color¶: Perform Color processing.

-p, --precision¶: Perform HiPrecision processing to result in NOPROJ.cub that might be processed through HiJACK.

-j, --jack¶: Do not test if HiJACK is needed, but force HiJACK to run.

--max_workers <max_workers>¶: This program uses multiprocessing to parallelize some of the work. This multiprocessing will default to use the number of processors on the host machine. If you want to throttle this to use less resources on your machine, indicate the number of processors you want to use.

--conf_dir <conf_dir>¶: Directory where configuration files for each of the hiproc algorithms are. Defaults to the configuration files distributed with the library.

bitflips¶

Deal with stuck bits in HiRISE pixels.

If there is some process which is flipping one (or more) of the HiRISE bits in a 14-bit pixel the wrong way, it will inadvertently add or subtract a certain DN value to a pixel. Naturally, it goes from 2^13 (8192) all the way down to 2.

In practice, these ‘bit-flipped’ pixels show up as spikes in the image histogram at DN values approximately centered on the image Median plus or minus the bit-flip value (8192, 4096, etc.). These are not narrow spikes and have variable width, so care must be taken when dealing with them. The large bit-flip values often show up as easily-identifiable islands in the DN histogram, but small bit-flip values (2, 4, 8, etc.) are often within the variability of the data and it is more difficult to cleanly isolate them.

This library and program deals with two aspects of the bit-flip problem: identification and mitigation.

Identification¶

This can be as simple as deciding that some bit-flip position (maybe 16 or 32) is the cut-off, and simply saying that the DN window of ‘good’ pixles is Median plus/minus the cut-off. However, this is often a blunt instrument, and can still leave problematic outliers.

An improved strategy is to attempt to identify the ‘islands’ in the DN histogram that are centered on the bit-flip positions, and attempt to find the gaps between those islands. Again, picking a bit-flip value cut-off, but this time selecting the low-point between two bit-flip-value-centered islands as the boundaries of the good-DN-window. This strategy works okay for test data, but real data of the martian surface often has complicated histograms, functions for this approach can be found un bitunflip.py.

The best strategy we have developed so far is to take the above one step further and treat the histogram curve as its own signal, identify all of the minima in the histogram, along with a sense of the standard deviation of the ‘good’ data in the image, and attempt to smartly pick the bounds of the good-DN-window.

Mitigation¶

This library and the program can perform one of two general activities to mitigate the bit-flip pixels once they have been identified.

Unflipping: It can attempt to un-flip the bits in the pixel to attempt to return it to what its ‘original’ DN value might have been. This is limited by however you select the bit-flip cut-off. Since you can’t identify below that cut-off, you also really can’t unflip below that cut-off. Functions for this approach can be found in bitunflip.py
Masking: This mitigation strategy simply sets these pixels to an arbitrary DN value or the ISIS NULL value. This basically just exercises the ‘identification’ part of this library and leaves it up to other image processing activities to decide how to deal with this data.

usage: bitflips [-h] [-v] [--logfile LOGFILE] [-k] [--version] [-o OUTPUT]
                [-w WIDTH] [--line] [-r REPLACEMENT] [-p]
                [--saveplot [SAVEPLOT]] [-n]
                file

file¶: ISIS Cube file or PDS IMG to clean.

-h, --help¶: show this help message and exit

-v, --verbose¶: Displays additional information as processing progresses.

--logfile <logfile>¶: The log file to write log messages to in addition to the terminal.

-k, --keep¶: Normally, the program will clean up any intermediary files, but if this option is given, it won’t.

--version¶: Show hiproc version number.

-o <output>, --output <output>¶: The name of the output .cub file to write. Optionally, if it starts with a ‘.’ it is considered a suffix and the input file will have its suffix removed, and this will be added input files. Default: .bitflip.cub

-w <width>, --width <width>¶: The number of medstd widths for bit-flip cleaning. (default: 5)

--line¶: Performs statistics along the line direction instead of column for the image area.

-r <replacement>, --replacement <replacement>¶: By default, the program will replace identified pixels with an appropriate NULL data value, but if provided this value will be used instead.

-p, --plot¶: Displays interactive plot for each area.

--saveplot <saveplot>¶: Saves plot for each area to a file. If a directory is provided it will be used to save the plots to, otherwise the directory of the input file will be used.

-n, --dryrun¶: Does not produce a cleaned output file.

EDR_Stats¶

Create an ISIS cube from a HiRISE EDR .img file and record some statistics.

These are the functionalities that the EDR_Stats pipeline performs that are reproduced here:

Convert a HiRISE EDR Product to an ISIS cube for subsequent pipeline processing using the ISIS system for much of the work.
Gather image statistics about the observation and write them out to a .json file for other programs to use.
Calculate the Signal-to-Noise Ratio (SNR).

In general, this program is the first, simple step of the HiRISE processing chain, and prepares data for the next step: HiCal.

Data Flow¶

Input Products:

Any HiRISE PDS EDR, typically ending in .img.

Output Products:

A .cub file for each .img file processed.
A .json file for each .img file processed containing summary
information about the image.

usage: EDR_Stats [-h] [-v] [--logfile LOGFILE] [-k] [--version] [-o OUTPUT]
                 [--db DB] [--histmin HISTMIN] [--histmax HISTMAX] [-g GAINS]
                 [--max_workers MAX_WORKERS]
                 some.img-file [some.img-file ...]

some.img-file¶: More than one can be listed here.

-h, --help¶: show this help message and exit

-v, --verbose¶: Displays additional information as processing progresses.

--logfile <logfile>¶: The log file to write log messages to in addition to the terminal.

-k, --keep¶: Normally, the program will clean up any intermediary files, but if this option is given, it won’t.

--version¶: Show hiproc version number.

-o <output>, --output <output>¶: Output filename. Optionally, if it starts with a ‘.’ it is considered a suffix and will be swapped with the inputfile’s suffix to determine the file to write. Default: .EDR_Stats.cub

--db <db>¶: The .json file to output. Optionally, if it starts with a ‘.’ it is considered a suffix and will be swapped with the input file’s suffix to determine the file to write. Default: .HiCat.json

--histmin <histmin>¶: The minimum percentage above which DN are counted. This only affects the value of STD_DN_LEVELS in the output .json file. If it is lower, more DN will be counted. Default: 0.01

--histmax <histmax>¶: The minimum percentage below which DN are counted. See –histmin for more information. Default: 99.99

-g <gains>, --gains <gains>¶: Path to the gains config PVL file. Defaults to EDR_Stats_gains_config.pvl distributed with the library.

--max_workers <max_workers>¶: If more than one image is provided to work on, this program will engage multiprocessing to parallelize the work. This multiprocessing will default to the number of processors on the machine. If you want to throttle this to use less resources on your machin, indicate the number of processors you want to use.

fft_clean¶

Perform fast-fourier-transform cleaning on HiRISE Channels.

usage: fft_clean [-h] [-v] [--logfile LOGFILE] [-k] [--version] [-o OUTPUT]
                 file

file¶: ISIS Cube file to clean.

-h, --help¶: show this help message and exit

-v, --verbose¶: Displays additional information as processing progresses.

--logfile <logfile>¶: The log file to write log messages to in addition to the terminal.

-k, --keep¶: Normally, the program will clean up any intermediary files, but if this option is given, it won’t.

--version¶: Show hiproc version number.

-o <output>, --output <output>¶: The name of the output .cub file to write. Optionally, if it starts with a ‘.’ it is considered a suffix and the input file will have its suffix removed, and this will be added to input file name. Default: .fftclean.cub

HiBeautify¶

Creates special color products (“pretty pictures”).

This creates a synthetic RGB product and an enhanced IRB product that are cosmetically improved over the raw/normalized color products. HiBeautify takes the .hicolornorm.pvl file (created by HiColorNorm) as its input. It uses the HiColorNorm cubes (extension _COLOR4.cub and _COLOR5.cub).

Data Flow¶

Input Products:

COLOR4.HiColorNorm and COLOR5.HiColorNorm files which are the
result of HiColorNorm.

Output Products:

IRB and RGB cubes

usage: HiBeautify [-h] [-v] [--logfile LOGFILE] [-k] [--version]
                  [-o_irb OUTPUT_IRB] [-o_rgb OUTPUT_RGB] [-c CONF]
                  HiColorNorm-cube HiColorNorm-cube

hicolornorm-cube¶: The COLOR4.HiColorNorm.cub and COLOR5.HiColorNorm.cub files.

-h, --help¶: show this help message and exit

-v, --verbose¶: Displays additional information as processing progresses.

--logfile <logfile>¶: The log file to write log messages to in addition to the terminal.

-k, --keep¶: Normally, the program will clean up any intermediary files, but if this option is given, it won’t.

--version¶: Show hiproc version number.

-o_irb <output_irb>, --output_irb <output_irb>¶: The filename to be used for the output IRB cube. If it begins with an underscore (‘_’) it will be assumed to be a suffix that will be appended to a name derived from the observation. Default: _IRB.cub

-o_rgb <output_rgb>, --output_rgb <output_rgb>¶: The filename to be used for the output RGB cube. If it begins with an underscore (‘_’) it will be assumed to be a suffix that will be appended to a name derived from the observation. Default: _RGB.cub

-c <conf>, --conf <conf>¶: Path to the HiBeautify config file. Defaults to HiBeautify.conf distributed with the library.

HiCal¶

Generate radiometrically corrected HiRISE Channel products.

These are the functionalities that the EDR_Stats pipeline performs that are reproduced here:

Perform a radiometric calibration correction and conversion to “I/F” units using the ISIS hical program.
Perform furrow correction. The image columns at the channel join are checked for furrows. Pixels in the furrow region (at the channel join) whose DN values have gone above a threshold are set to the null pixel value as these pixels can not be calibrated. If a RED-filter image experiences furrowing, a rare event usually caused by an improperly commanded observation, then the furrowed pixels will be permanently set to null pixels for all of the HiRISE standard and extras products. For BG and IR-filter images that make up the color products, the furrowed pixels will be interpolated in the HiColorNorm pipeline step.
If an observation is determined to have furrows then an entry with the key “zapped” is set to “true” in the output .json file.
Due to HiRISE instrument instability problems, the HiCal pipeline performs a noise reduction procedure to reduce the number of bad pixels in an image observation. The noise correction is applied when the standard deviation of the dark pixel or mask regions exceed a threshold, or if the number of LIS pixels exceeds a threshold. A newer version of that processing can be engaged if –bitflipwidth is set to a non-zero value.
Sometimes operating at lower temperatures can result in LIS pixels in the calibration areas, but not in the image area that can result in an erroneous calibration. Some new processing to deal with that can be engaged if –lisfix is set to a value less than 1.
A high-pass filter “cubenorm” step is applied to the calibrated image. Due to camera instabilities, residual vertical striping often exists in the imaging that is corrected by this empirical method. The average standard deviation of this change is calculated and stored in the .json file.
The ISIS program “hidestripe” is applied to suppress horizontal stripping seen whenever an observation is acquired using mixed-binning commanding. The standard deviation of this change is calculated and stored in the .json file.

HiCal is the second step in the HiRISE processing chain, after EDR_Stats and results in data that is read for the next step: HiStitch. If you are running HiCal individually without all EDRs from the Observation in the same directory, please be aware of the –bin2 and –bin4 options.

Data Flow¶

Input Products:

A .cub file that has been converted from an EDR.
A .json file that contains summary information about the .cub file.

Output Products:

A calibrated .cub file for each input .cub file processed.
Modifies each provided .json file.

usage: HiCal [-h] [-v] [--logfile LOGFILE] [-k] [--version] [-o OUTPUT]
             [-c CONF] [--db DB] [-b BITFLIPWIDTH] [-t LIS_TOLERANCE]
             [--nfconf NFCONF] [--bin2] [--nobin2] [--bin4] [--nobin4] [-n]
             [--max_workers MAX_WORKERS]
             cube_files) [cube_file(s ...]

cube_file(s)¶: More than one can be listed here.

-h, --help¶: show this help message and exit

-v, --verbose¶: Displays additional information as processing progresses.

--logfile <logfile>¶: The log file to write log messages to in addition to the terminal.

-k, --keep¶: Normally, the program will clean up any intermediary files, but if this option is given, it won’t.

--version¶: Show hiproc version number.

-o <output>, --output <output>¶

-c <conf>, --conf <conf>¶: Configuration file to use, Defaults to HiCal.conf distributed with the library.

--db <db>¶: The .json file to use. Optionally, if it starts with a ‘.’ it is considered an extension and will be swapped with the input file’s extension to find the .json file to use. Default: .HiCat.json

-b <bitflipwidth>, --bitflipwidth <bitflipwidth>¶: The number of medstd widths for bit-flip cleaning. Leaving it unset will result in the setting being determined by the value of LIS_Mask in the configuration file. Explicitly setting it to zero implies no bit-flip cleaning and will use the old ANALYZECUBENORMSTATS behavior. Explicitly setting it to 5 is identical to the BITFLIPS setting.

-t <lis_tolerance>, --lis_tolerance <lis_tolerance>¶: If the fraction of LIS pixels in each of the Reverse-Clock and Buffer areas are higher than this value, lis-fix processing will occur for that area. Leaving it unset will result in the value being determined by the value of LIS_Fix in the configuration fie. Explicitly setting it to 0.4 is the same as LIS_Fix = TRUE, and explicitly setting it to 1.0 is the same as LIS_Fix = FALSE.

--nfconf <nfconf>¶: Defaults to NoiseFilter.conf distributed with the library.

--bin2¶: By default HiCal will look at all of the HiRISE .cub files in the directory with the input file(s) in it, and determine whether any of the EDRs from this observation was bin2 and will internally take care of setting this switch, but if you’re just processing some RED channels and not all of them, then although IR10 may be binned, and you didn’t convert it to a .cub file or even download it, then it isn’t there for this program to find, and explicitly setting this switch is needed.

--nobin2¶: Forcibly sets the –bin2 switch to off, even if it would otherwise be automatically determined to be on.

--bin4¶: Same as the –bin2 switch, but for setting whether any EDRs from this observation are bin4.

--nobin4¶: Forcibly sets the –bin4 switch to off, even if it would otherwise be automatically determined to be on.

-n, --info¶: If given, the program will not run, but will instead report information about the input files and various things that the program would have done, if it was run regularly.

--max_workers <max_workers>¶: If more than one image is provided to work on, this program will engage multiprocessing to parallelize the work. This multiprocessing will default to the number of processors on the machine. If you want to throttle this to use less resources on your machin, indicate the number of processors you want to use.

HiccdStitch¶

Stitch together the same HiRISE color-filter CCDs from an observation to create a single mosaicked image file.

This program:

Optionally performs cubenorm processing on each CCD using a training area
that has been selected by the user. Cubenorm processing is required whenever radiometric problems, such as vertical striping, remain after the radiometric calibration step in the HiCal Pipeline. The cubenorm processing is described in the ISIS program cubenorm.
Balance the CCD products, created by HiStitch, to radiometrically match.
A multiplicative constant is applied to each CCD to force the overlapping areas of adjacent CCDs to be identical. The resulting balanced CCD products, have the naming convention *.balance.cub. These products are used by subsequent pipeline processing for creating color products, RDR products, and DTM products.
Join (stitch together) the CCD products to form an image of the entire
observation.

HiccdStitch is the fourth step in the HiRISE processing chain, after HiStitch. If an observation has RED, IR and BG CCDs, then this program would need to be run once for each set.

Data Flow¶

Input Products:

.cub files of the same color (RED, IR, or BG) which are the result of
HiStitch.
.json files that start with the CCD ID of each source .cub file.

Output Products:

A stitched .cub file for that color.
A .balance.cub file for each input cube file.
A .json file with summary information about the stitched image.

usage: HiccdStitch [-h] [-v] [--logfile LOGFILE] [-k] [--version] [-o OUTPUT]
                   [-c CONF] [--db DB] [--sline SLINE] [--eline ELINE]
                   [--cubenorm CUBENORM [CUBENORM ...]]
                   cub-file [cub-file ...]

cub-file¶: Cubes to assemble, which are presumably the output of HiStitch. They must all be from the same detectors, so either all RED, all IR, or all BG cubes.

-h, --help¶: show this help message and exit

-v, --verbose¶: Displays additional information as processing progresses.

--logfile <logfile>¶: The log file to write log messages to in addition to the terminal.

-k, --keep¶: Normally, the program will clean up any intermediary files, but if this option is given, it won’t.

--version¶: Show hiproc version number.

-o <output>, --output <output>¶: The name of the output .cub file to write. Optionally, if it starts with a ‘.’ it is considered a suffixand will be added to the Observation ID of the input files. Default: .HiccdStitch.cub

-c <conf>, --conf <conf>¶: Path to the HiccdStitch config file. Defaults to HiccdStitch.conf distributed with the library.

--db <db>¶: The .json file to output. Optionally, if it starts with a ‘.’ it is considered an extension and will be swapped with the output file’s extension to determine the .json filename to use. Default: .HiCat.json

--sline <sline>¶: If given, will be used as the starting line to crop the image at in order to create a training area to use for cubenorm processing.

--eline <eline>¶: If given, will be used as the ending line for cubenorm cropping, see –sline for more information.

--cubenorm <cubenorm>¶: To engage cubenorm processing a list of true or false values (could be 0 or 1) that must match the number of input cubesmust be given to indicate which of the input cubes should have cubenorm processing applied. If you only had four inputcubes, then --cubenorm 0 0 1 1 would not run cubenorm processing on the first two cubes, but would run it on the lasttwo, etc. The default is not to run this processing on any.

HiColorInit¶

Prepares an observation for color co-registration and processing.

Prepares images for coregistration. It takes the HiStitch CCD product balance cubes and scales the BG and IR CCDs to match the corresponding RED CCDs. If it can, the program deals with these two sets:

RED4 - BG12 - IR10
RED5 - BG13 - IR11

HiColorInit must happen after HiccdStitch.

Data Flow¶

Input Products:

balance.cub files for the three or six of the CCDs lsited above, which
are the result of HiccdStitch.

Output Products:

A .precolor.cub file for each of the BG and IR input balance.cub files.

usage: HiColorInit [-h] [-v] [--logfile LOGFILE] [-k] [--version]
                   [-o OUTPUT_SUFFIX]
                   balance.cub-files [balance.cub-files ...]

balance.cub-files¶: Either one or both sets of RED/IR/BG .balance.cub files. However, that’s tedious to type, so you could just type in *.balance.cub here, and the program will sort out what it needs.

-h, --help¶: show this help message and exit

-v, --verbose¶: Displays additional information as processing progresses.

--logfile <logfile>¶: The log file to write log messages to in addition to the terminal.

-k, --keep¶: Normally, the program will clean up any intermediary files, but if this option is given, it won’t.

--version¶: Show hiproc version number.

-o <output_suffix>, --output_suffix <output_suffix>¶: The input color (BG & IR) color cubes will have their final suffix removed, and this suffix added. Default: .precolor.cub

HiColorNorm¶

Normalizes HiRISE Color products.

This program will normalize the BG and IR color across the left and right halves of each set of color CCDs. It uses the HiSlither COLOR4 and COLOR5 cubes.

Data Flow¶

Input Products:

COLOR4 and COLOR5 files which are the result of HiSlither.

Output Products:

creates UNFILTERED versions of the COLOR4 and COLOR5 files.
creates HiColorNorm versions of the COLOR4 and COLOR5 files.

usage: HiColorNorm [-h] [-v] [--logfile LOGFILE] [-k] [--version] [-o OUTPUT]
                   [-c CONF] [-n]
                   COLOR-cube COLOR-cube

color-cube¶: The COLOR4.cub and COLOR5.cub files

-h, --help¶: show this help message and exit

-v, --verbose¶: Displays additional information as processing progresses.

--logfile <logfile>¶: The log file to write log messages to in addition to the terminal.

-k, --keep¶: Normally, the program will clean up any intermediary files, but if this option is given, it won’t.

--version¶: Show hiproc version number.

-o <output>, --output <output>¶: The filename to be used for the output color cube. If it begins with an underscore (‘_’) it will be assumed to be a suffix that will be appended to a name derived from the observation. Default: _COLOR.cub

-c <conf>, --conf <conf>¶: Path to the HiColorNorm config file. Defaults to HiColorNorm.conf distributed with the library.

-n, --nounfiltered¶: Stops creation of an unfiltered cube.

HiJACK¶

Resamples images from individual CCDs to ideal camera geometry and mosaicks them into a single cube.

Data Flow¶

Input Products:

All color balance.cub files which are the result of HiccdStitch.

Output Products:

regdef.pvl, flat.tab and control.pvl files relating all files to RED4 or 5
smear and jitter files from resolve_jitter
jittery.bc file from ISIS hijitter
dejittered.cub from ISIS hijitter
The following dejittered noproj mosaics:
- _RED4-5.NOPROJ.cub
- RED.NOPROJ.cub
- _IR.NOPROJ.cub
- _BG.NOPROJ.cub
- _IRB.NOPROJ.cub

usage: HiJACK [-h] [-v] [--logfile LOGFILE] [-k] [--version] [-o OUT_DIR]
              [-c CONF_DIR] [-n] [-b BASE_CCD_NUMBER] [--plotflats]
              balance.cub-files [balance.cub-files ...]

balance.cub-files¶: All of the .balance.cub files created by HiccdStitch.

-h, --help¶: show this help message and exit

-v, --verbose¶: Displays additional information as processing progresses.

--logfile <logfile>¶: The log file to write log messages to in addition to the terminal.

-k, --keep¶: Normally, the program will clean up any intermediary files, but if this option is given, it won’t.

--version¶: Show hiproc version number.

-o <out_dir>, --out_dir <out_dir>¶: The directory where HiJACK output will be placed. If it doesn’t already exist, it will be created. Default: ./HiJACK

-c <conf_dir>, --conf_dir <conf_dir>¶: Directory where ResolveJitter.conf and HiJACK.conf can be found. Defaults to the directory distributed with the library.

-n, --noplot¶: Will stop the display of before and after jitter plots.

-b <base_ccd_number>, --base_ccd_number <base_ccd_number>¶: The CCD number that will be used in a variety of ways to set parameters for the resulting image. Default: 5

--plotflats¶: If given, HiJACK will expect two file paths which should be the prehijack and posthijack flat files created by a previous run of HiJACK. This only makes a plot, it does not engage any HiJACK processing again.

HiJitReg¶

HiJitReg registers color CCDs to corresponding red CCDs by using the ISIS tool hijitreg to perform a deconvolution of jittered image data.

This program corrects for spacecraft jitter and prepares images for coregistration. Using the ISIS program hijitreg it creates a reference grid of control points (extension .control.pvl) from the RED product, and then attempts to locate those points within the BG and IR products.

The configuration file for HiJitReg (HiJitReg.conf) describes the control point grid density (currently it’s a 4-column, 200-line grid) and correlation tolerance. It also specifies the sizes of the search window and pattern window. The pattern window “walks” through the search window in order to locate the local maximum. The calculated translation is then recorded in the control net file for slithering.

Some adaptability is built-in to HiJitReg. It will add columns if a channel is missing, and will triple the grid point density if less than 25% of the points register on the first pass. It will also increase the size of the search window if more than two points have the edge of their pattern box close to or beyond the edge of the search box (“edgy” points).

It also uses a smoothing algorithm to ignore points that are out-of-bounds or are poorly registered. A JitterPlot is created showing the results.

HiJigReg also works on one or both of the “color” sets:

RED4 - BG12 - IR10
RED5 - BG13 - IR11

HiJitReg must be run after HiColorInit, and it creates a .hislither.pvl file which is then submitted to the HiSlither_Pipeline.

Data Flow¶

Input Products:

RED4 and 5 balance.cub files which are the result of HiccdStitch.
BG and IR precolor.cub files which are the result of HiColorInit.

Output Products:

creates regdef.pvl and flat.tab and control.pvl files for each
BG and IR cube provided.

usage: HiJitReg [-h] [-v] [--logfile LOGFILE] [-k] [--version] [-c CONF]
                balance.cub and balance.precolor.cub files
                [balance.cub and balance.precolor.cub files ...]

balance.cub and balance.precolor.cub files¶: Either one or both sets of RED .balance.cub and IR/BG .balance.precolor.cub files. However, that’s tedious to type,so you could just type in *.balance*cub here, and the program will sort out what it needs.

-h, --help¶: show this help message and exit

-v, --verbose¶: Displays additional information as processing progresses.

--logfile <logfile>¶: The log file to write log messages to in addition to the terminal.

-k, --keep¶: Normally, the program will clean up any intermediary files, but if this option is given, it won’t.

--version¶: Show hiproc version number.

-c <conf>, --conf <conf>¶: Path to the HiJitReg config file. Defaults to HiJitReg.conf distributed with the library.

HiNoProj¶

Resamples images from individual CCDs to ideal camera geometry and mosaicks them into a single cube.

Data Flow¶

Input Products:

RED balance.cub files which are the result of HiccdStitch.

Output Product:

NOPROJ.cub

usage: HiNoProj [-h] [-v] [--logfile LOGFILE] [-k] [--version] [-o OUTPUT]
                [-c CONF] [-b BASE_CCD_NUMBER]
                balance.cub-files [balance.cub-files ...]

balance.cub-files¶: The RED .balance.cub files created by HiccdStitch.

-h, --help¶: show this help message and exit

-v, --verbose¶: Displays additional information as processing progresses.

--logfile <logfile>¶: The log file to write log messages to in addition to the terminal.

-k, --keep¶: Normally, the program will clean up any intermediary files, but if this option is given, it won’t.

--version¶: Show hiproc version number.

-o <output>, --output <output>¶: The filename to be used for the output noproj cube. If it begins with an underscore (‘_’) it will be assumed to be a suffix that will be appended to a name derived from the observation. Default: _RED.NOPROJ.cub

-c <conf>, --conf <conf>¶: Path to the HiNoProj config file. Defaults to HiNoProj.conf distributed with the library.

-b <base_ccd_number>, --base_ccd_number <base_ccd_number>¶: The CCD number that will be given to the MATCH parameter of ISIS noproj. Default: 5

HiPrecisionInit¶

Examines slither files to determine whether HiJACK processing is needed, or if HiNoProj is sufficient.

If the average of the absolute difference between the slither fit and the measurements in the slither.txt file is greater than the threshhold in the configuration file, this program will tell you that.

usage: HiPrecisionInit [-h] [-v] [--logfile LOGFILE] [-k] [--version]
                       [-c CONF]
                       slither.txt-files [slither.txt-files ...]

slither.txt-files¶: The text files created by HiSlither.

-h, --help¶: show this help message and exit

-v, --verbose¶: Displays additional information as processing progresses.

--logfile <logfile>¶: The log file to write log messages to in addition to the terminal.

-k, --keep¶: Normally, the program will clean up any intermediary files, but if this option is given, it won’t.

--version¶: Show hiproc version number.

-c <conf>, --conf <conf>¶: Path to the HiPrecisionInit config file. Defaults to HiPrecisionInit.conf distributed with the library.

HiSlither¶

HiSlither uses the output of HiJitReg to perform the registration of color CCDs to red CCDs and output a color cube.

The HiSlither pipeline does a “rubber-sheet stretch” of the IR and BG to correct for jitter and match the RED based on the control net output (extension .control.pvl) of HiJitReg. It then stacks the IR, RED, and BG and on top of one another (in IR-RED-BG order).

The ISIS program slither computes an image transformation that shifts lines and samples as needed according to the control net file (extension .control.pvl) created by HiJitReg. It also creates a text file (extension .slither.txt) containing the transformation statistics, and a text file (extension .control.csv) from the smoothing algorithm that checks for “bad” (those that may cause misregistration) control points. The transformation for each of the valid control points in a row is averaged together to arrive at a transformation for the entire row (line). Between rows, the transformation is interpolated.

The ISIS program hicubeit merges the IR, RED, and BG cubes on top of one another to create two separate 3-band color products (extension _COLOR4.cub and _COLOR5.cub).

The ISIS program trim removes the inner overlap from the color halves.

Data Flow¶

Input Products:

RED4 and 5 balance.cub files which are the result of HiccdStitch.
BG and IR precolor.cub files which are the result of HiColorInit.
The regdef.pvl, flat.tab, and control.pvl files created by HiJitReg.

Output Products:

creates a slither.txt and slither.cub file for each of the color
inputs.
creates COLOR4.cub and COLOR5.cub files which are the multi-band
aligned images that match RED4 - BG12 - IR10 and RED5 - BG13 - IR11.

usage: HiSlither [-h] [-v] [--logfile LOGFILE] [-k] [--version]
                 RED balance.cub and color balance.precolor.cub files
                 [RED balance.cub and color balance.precolor.cub files ...]

red balance.cub and color balance.precolor.cub files¶: Either one or both sets of RED .balance.cub and IR/BG .balance.precolor.cub files. However, that’s tedious to type,so you could just type in *.balance*cub here, and the program will sort out what it needs.

-h, --help¶: show this help message and exit

-v, --verbose¶: Displays additional information as processing progresses.

--logfile <logfile>¶: The log file to write log messages to in addition to the terminal.

-k, --keep¶: Normally, the program will clean up any intermediary files, but if this option is given, it won’t.

--version¶: Show hiproc version number.

HiStitch¶

Stitch together two HiRISE channel files to create a single CCD image file.

These are the functionalities that the HiStitch pipeline performs that are reproduced here:

Join (or stitch) together two channel products to form a whole CCD product.
The channel 1 product is on the left, channel 0 on the right.
When joining the two channel products, the ISIS histitch program first
obtains the image average of the two channel products at the join area. The program then adjusts the average of channel 1 to match the average of channel 0 through a multiplicative correction. This step produces a radiometrically seamless CCD product.
For Bin 2 and 4 images a fix is applied to images that have experienced
furrowing. If furrowing has been detected, a highpass/lowpass filter is applied to the furrow region for those pixels that have not been set to the null pixel value in the HiCal step.

HiStitch is the third step in the HiRISE processing chain, after HiCal and results in data that is read for the next step: HiccdStitch. If you are running HiCal individually without all EDRs from the Observation in the same directory, please be aware of the –bin2 and –bin4 options.

Data Flow¶

Input Products:

Two .cub files from the same CCD that have been processed through HiCal.
Two .json files that contain summary information about the .cub files.

Output Products:

A stitched .cub file.
A .json file with summary information about the stitched image.

usage: HiStitch [-h] [-v] [--logfile LOGFILE] [-k] [--version] [--db DB]
                [--db2 DB2] [--dbout DBOUT] [-o OUTPUT] [-c CONF]
                cube0.cub-file [cube1.cub-file]

cube0.cub-file¶

cube1.cub-file¶

-h, --help¶: show this help message and exit

-v, --verbose¶: Displays additional information as processing progresses.

--logfile <logfile>¶: The log file to write log messages to in addition to the terminal.

-k, --keep¶: Normally, the program will clean up any intermediary files, but if this option is given, it won’t.

--version¶: Show hiproc version number.

--db <db>¶: The .json file to use. Optionally, if it starts with a ‘.’ it is considered an extension and will be swapped with the input file’s extension to find the .json file to use. Default: .HiCat.json

--db2 <db2>¶: The second .json file to use. Optionally, if it starts with a ‘.’ it is considered a suffix and will be swapped with the second input file’s suffix to find the .json file to use. Default: .HiCat.json

--dbout <dbout>¶: The name of the output .json file to write. Optionally, if it starts with a ‘.’ it is considered a suffix and will be added to the CCD ID of the observation of the input files. Default: .HiCat.json

-o <output>, --output <output>¶: The name of the output .cub file to write. Optionally, if it starts with a ‘.’ it is considered a suffixand will be added to the CCD ID of the observation of the input files. Default: .HiStitch.cub

-c <conf>, --conf <conf>¶: Path to the HiStitch config file. Defaults to HiStitch.conf distributed with the library.

JitPlot¶

JitPlot converts the files that HiJitReg creates and displays a plot which represents them.

usage: JitPlot [-h] [-v] [--logfile LOGFILE] [-k] [--version] [-c CONF]
               balance.precolor.cub files [balance.precolor.cub files ...]

balance.precolor.cub files¶: Either one or both sets of RED .balance.cub and IR/BG .balance.precolor.cub files. However, that’s tedious to type,so you could just type in *.balance*cub here, and the program will sort out what it needs.

-h, --help¶: show this help message and exit

-v, --verbose¶: Displays additional information as processing progresses.

--logfile <logfile>¶: The log file to write log messages to in addition to the terminal.

-k, --keep¶: Normally, the program will clean up any intermediary files, but if this option is given, it won’t.

--version¶: Show hiproc version number.

-c <conf>, --conf <conf>¶: Path to the HiJitReg config file. Defaults to HiJitReg.conf distributed with the library.

lisfix¶

Converts LIS values in reverse-clock and buffer areas to derived values.

For a variety of reasons, but highlighted with the August 2020 change of ADC settings, the HiRISE LUT process can clip the DN histogram resulting in good data in the image area, but mostly LIS values in the reverse-clock and buffer areas, which are crucial for calibration. This program attempts to replace “missing” data in those areas with values derived from the ramp area, the dark area, and models of expected reverse-clock medians.

usage: lisfix [-h] [-v] [--logfile LOGFILE] [-k] [--version] [-o OUTPUT] file

file¶: ISIS Cube file.

-h, --help¶: show this help message and exit

-v, --verbose¶: Displays additional information as processing progresses.

--logfile <logfile>¶: The log file to write log messages to in addition to the terminal.

-k, --keep¶: Normally, the program will clean up any intermediary files, but if this option is given, it won’t.

--version¶: Show hiproc version number.

-o <output>, --output <output>¶

SlitherStats¶

SlitherStats computes statistics from the results of HiSlither.

usage: SlitherStats [-h] [-v] [--logfile LOGFILE] [-k] [--version] [-p]
                    slither.txt files [slither.txt files ...]

slither.txt files¶

-h, --help¶: show this help message and exit

-v, --verbose¶: Displays additional information as processing progresses.

--logfile <logfile>¶: The log file to write log messages to in addition to the terminal.

-k, --keep¶: Normally, the program will clean up any intermediary files, but if this option is given, it won’t.

--version¶: Show hiproc version number.

-p, --plot¶: Display plots

resolve_jitter¶

Perform the jitter derivation for a HiRISE observation.

Outputs are the jitter in the x (sample) and y (line) directions in pixels, and the time (ephemeris time at that translation), average error between the derived jitter function and the original data, linerate (line time read from flat files), and TDI. These outputs are written to a text file. Another output is the pixel smear, also written to a file.

The C++ version of this runs ~5x faster, FYI.

WARNING: This program and its modules may not be fully functional and may return poor results.

usage: resolve_jitter [-h] [-v] [--logfile LOGFILE] [-k] [--version] [-c CONF]
                      [--csv] [--lineinterval LINEINTERVAL] [--outdir OUTDIR]
                      [--outprefix OUTPREFIX] [-p] [--saveplot [SAVEPLOT]]
                      [--whichmatch1] [--whichmatch2] [--whichmatch3]
                      [files [files ...]]

files¶: Three flat.txt files that are the output of ISIS hijitreg.

-h, --help¶: show this help message and exit

-v, --verbose¶: Displays additional information as processing progresses.

--logfile <logfile>¶: The log file to write log messages to in addition to the terminal.

-k, --keep¶: Normally, the program will clean up any intermediary files, but if this option is given, it won’t.

--version¶: Show hiproc version number.

-c <conf>, --conf <conf>¶: Path to a ResolveJitter.conf file, only needed if –lineinterval isn’t given.

--csv¶: This program writes out a fixed-width data file with extra information for plotting. This also writes out a comma-separated version.

--lineinterval <lineinterval>¶: The number of lines to use to set the number of Fourier transform intervals, defaults to Control_Lines in the ResolveJitter.conf file.

--outdir <outdir>¶: Output directory. Defaults to the directory of the first input file.

--outprefix <outprefix>¶: Prefix string for output files. If not given, will default to the Observation ID of the images.

-p, --plot¶: Displays interactive plot.

--saveplot <saveplot>¶: Saves plot to a default filename in the output directory. If a filename is provided it will be used to save the plot.

--whichmatch1¶: If specified, the sense of the offsets for the first file will be relative to the MATCH cube, rather than the FROM cube.

--whichmatch2¶: If specified, the sense of the offsets for the second file will be relative to the MATCH cube, rather than the FROM cube.

--whichmatch3¶: If specified, the sense of the offsets for the third file will be relative to the MATCH cube, rather than the FROM cube.

Console Programs¶

hiproc¶

bitflips¶

Identification¶

Mitigation¶

EDR_Stats¶

Data Flow¶

fft_clean¶

HiBeautify¶

Data Flow¶

HiCal¶

Data Flow¶

HiccdStitch¶

Data Flow¶

HiColorInit¶

Data Flow¶

HiColorNorm¶

Data Flow¶

HiJACK¶

Data Flow¶

HiJitReg¶

Data Flow¶

HiNoProj¶

Data Flow¶

HiPrecisionInit¶

HiSlither¶

Data Flow¶

HiStitch¶

Data Flow¶

JitPlot¶

lisfix¶

SlitherStats¶

resolve_jitter¶

hiproc

Navigation

Related Topics