Check and compare output files

When your container run command completes, you’ll want to check the output to ensure it ran to completion, and then look at the output to make sure it did what you expected it to do.

Ensure that your containerized workflow completed

You can determine if a Docker container completes when it disappears from the docker ps console output, and when an Apptainer container process has completed in the shell. When your Apptainer or Docker container process ends, examine the log output and determine if there are any noticeable error messages.

Error messages may not occur at the very end of the log file. The tutorial script did not include built in checks to determine if prior steps’ files exist before proceeding, so the script will attempt to run all commands in order regardless of whether the previous command completed successfully. This can mean that commands get run on output that doesn’t exist. Check the log and find the end of each output step to determine if the step completed successfully and confirm that all steps in the workflow worked.

If you have a longer workflow, or steps that spit out a lot of output, it can be hard to find where one workflow step ended and the next one began. An enhancement you can make to your run_workflow.sh script can be to output each step to the console and then run it, like this:

# Output the name of the step you are running
echo "Running fslroi"

# Create a string variable with the command text and inputs
cmd="fslroi ${scans_dir}/${session_label}/${dti_scan_id}/${dti_scan_filename} ${output_dir}/${session_label}_nodif.nii.gz 0 1"

# Output the full command you are going to run to the console without actually running it yet
echo "Running command:"
echo ${cmd}

# Use the bash "eval" command to run the exact same command
# the "die" command will end the script if this eval step fails.
eval ${cmd} || die "fslroi failed"

You should not only check the logs, but make sure that all the files in your output folder are the expected size and quantity that you should have as compared to your initial local testing of your workflow script.

Once you are sure all steps ran and you have the correct number of files, then you can compare the container output and local output files directly.

Comparing output data

If you determine that your container finished all the steps, generated the correct number of files and/or gave you some quantitative values that look like what you expect, you still want to check its work. It’s a good idea to programmatically compare the data generated by your container with your original test data so you can ensure that your container output matches your expected output.

Compare quantitative values

If your output is quantitative values - check whether they are identical. They may not be identical due to differing systems with certain programs, or differing program versions from your local system to the container. If you expect them to be the same and they are not, then begin to examine the steps that created those values and work backwards to try to determine where the containerized version diverged from your local version.

Compare image files

If your output is image files - examine your final outputs and find out whether they look the same visually and quantitatively. You can use an image viewer like fsleyes or freeview to check your images based on the file type.

The images may not be identical due to differing systems with certain programs, or differing program versions from your local system to the container. If you are expecting identical image outputs, and need to confirm that your local output is identical to your container output, you can compare them quantitatively with fslmaths.

Rename your files so you can know where each one came from, since you will be running commands using two files with the same name. First create a difference image of your local output minus your container output:

fslmaths OAS30001_MR_d3132_dtidata_FA_local.nii.gz -sub OAS30001_MR_d3132_dtidata_FA_container.nii.gz OAS30001_MR_d3132_dtidata_FA_local_minus_container.nii.gz

Use fslstats to determine if any data exists in the difference image:

fslstats -V OAS30001_MR_d3132_dtidata_FA_local_minus_container.nii.gz

If the output of the fslstats -V command is zero, then there is no data in the difference image, which means that the two images that went into the fslmaths command were identical. If the fslstats command gives non-zero output, then there is a difference between the two input images. You can view the difference image in an image viewer to see exactly where the images differed which might give some clues about where the discrepancy came from.

If your comparison output is not what you expect, use similar steps to check each prior command and work backwards until you can find the place where the local and containerized outputs diverged.