Using landmark correspondences in a multi-stage registration

[ishaanb92]

Hello,

Problem description:
I'm working on developing a deep learning based method to find landmark correspondences that can aid registration, especially for the registration of organs in motion (e.g.: lung). I'm working with the following lung CT dataset (https://med.emory.edu/departments/radiation-oncology/research-laboratories/deformable-image-registration/index.html).

In my setup, I use one affine registration and two b-spline registrations (https://elastix.lumc.nl/modelzoo/par0008/). My only edit to the parameter files is adding a "CorrespondingPointsEuclideanDistanceMetric" (weight=0.05) term in all the parameter files, and supplying landmarks correspondences predicted by deep learning model as an argument in the elastix command.

Question
The landmark correspondences are made between the original fixed and moving image domains. However, the registration quality (w.r.t. TRE calculated using manual landmarks) worsens when I use landmarks (compared to the registration with the unedited parameter files). It could be that my deep learning model predicts incorrect correspondences, however it would help in eliminating potential causes if the following question was clarified.

Does elastix internally transform the points when going from one stage to the next?

Let us say my original point-set was X_f and X_m. The first stage (affine) would use this set to estimate a transform T_a. For the next step (B-spline-1), does elastix still use X_f and X_m or T_a(X_f) and X_m? I understand this is how it works in terms of images, i.e., the resampled moving image from a previous stage is used as the input for the next stage.

I can imagine that alignment may suffer if the same point-set (X_f and X_m) was used for all three stages.

I hope my description and question are clear.

Cheers,
Ishaan

[ntatsisk]

Hi @ishaanb92, thanks for reaching out and sharing your project - very interesting!

I would suppose that elastix should also transform the corresponding points between each registration/stage - I will take a deeper look into it. Are you using the executable? Could you share the command you are using?

The landmark correspondences are made between the original fixed and moving image domains.

How do you make sure that this is the case? Do you post-process the corresponding points after the DL model? Because my initial hunch would be that the problem might lie somewhere there.

Also relevant:

• At the manual: section 4.2-> on how to define the points (index vs coordinate) and section 6.1.7-> a bit of theory on how they are used.
• I don't know if this is still true but the manual indicates that CorrespondingPointsEuclideanDistanceMetric needs to be at the end of the metric list (@N-Dekker?)
• Another example of the model zoo that uses corresponding points but not for every registration stage: https://elastix.lumc.nl/modelzoo/par0014/
• Jupyter example in case you would like to use python (maybe more convenient together with DL) : https://github.com/InsightSoftwareConsortium/ITKElastix/blob/main/examples/ITK_Example05_PointRegistration.ipynb

[ishaanb92]

Hi @ntatsisk,

Thanks for the prompt reply and your interest in the work! :)

Command:

How do you make sure that this is the case? Do you post-process the corresponding points after the DL model? Because my initial hunch would be that the problem might lie somewhere there.

My DL model accepts two inputs/images, predicts a fixed number of landmark candidates in each image (limited by GPU memory) and predicts correspondences between them. For each patient, I give my DL model, the fixed (inhale) and moving (exhale) images and post-process the the landmark correspondences into a .txt file in accordance with the transformix format. The landmark correspondences are between voxel indices but I convert them to physical (world) coordinates.

I also did a bit of digging into the elastix.log when the 1st B-spline registration (2nd registration stage) starts:

At the start of registration, it is (re-)reading the points files, but I do not see a statement indicating a transformation using an earlier stage. I understand this is not conclusive either way, and it may be transformed internally.

Thanks for the reference parameter file, I'll take a look. If anything else needs clarification, let me know.

Thanks again for your help! :)

Cheers,
Ishaan

[ntatsisk]

Thanks @ishaanb92, for digging into this further. Indeed, the "Reading landmark file" line that is repeated more than once looked suspicious in the beginning so I did some extra tests. My current conclusion is that it is simply redudant, and in fact the registration works as expected.

You can test on your data by setting ("Metric0Weight" "0.0") and ("Metric1Weight" "1.0"). That way only the CorrespondingPointsEuclideanDistanceMetric is taken into account for the registration, while both metrics are reported. I did that for a two stage (affine + affine) registration and I attach the log here: log_affine_affine.txt. As you can see "Metric1" converges to 2.365649 with no "jump" between the two stages. Hence, I assume there is no problem of the points not being transformed properly. Let me know if you try it and the same behavior is observed in your case.

I could look into it a bit further:

1. Can you send me the fixed_landmarks_elx.txt and moving_landmarks_elx.txt files that you are using? Ideally, also the corresponding files before the post-processing (raw points that come out of the model).
2. I downloaded the copd1_eBHCT and copd1_iBHCT from the DIRLab website. However, I am having trouble converting the files from .img to .mha as you did. Can you give me some directions on that?

[ishaanb92]

Hi @ntatsisk,

Thanks for looking into this! I checked the log, I see that the transition of the CorrespondingPointsEuclideanDistanceMetric from one stage to the other is smooth, with no jump.

All the files needed to reproduce the registration can be found here : https://surfdrive.surf.nl/files/index.php/s/kpUw3dicV1mWttU

I do not save the intermediate points, my DL model gives me a list of landmark correspondences that I save as elastix compliant .txt files.

Thanks once more for looking into this and I think I am convinced that elastix transforms the points internally, based on the "smoothness" of the metric across different registration stages.

Cheers,
Ishaan