Why can't the original camera position optimization optimize the pose and instead renders a distorted image?

[harry-tang89]

Problem: When using the official project (tutorials/camera_position_optimization_with_differentiable_rendering.ipynb) to optimize the camera pose, the rendered image doesn't match the expected result, and the loss doesn't decrease as expected. Instead, the image looks strange (the rendered image is visually distorted).

• When I use the following code with blur_radius=np.log(1. / 1e-4 - 1.) * blend_params.sigma in the RasterizationSettings, the rendered image looks distorted, as shown below:

• However, when I set blur_radius = 0, the rendered image looks more reasonable:

But in this case, the loss increases continuously, which is not what I expect.

• The original mesh (in the current camera pose) looks like this:

Code Explanation:

# Parameters for blending and rasterization
blend_params = BlendParams(sigma=1e-4, gamma=1e-4)

# Set up rasterization settings
raster_settings = RasterizationSettings(
image_size=image_size.cpu().tolist(),
blur_radius=np.log(1. / 1e-4 - 1.) * blend_params.sigma,  # Derived from sigma
faces_per_pixel=100,  # Number of faces considered per pixel
)

# Create the renderer with the mesh rasterizer and shader
renderer = MeshRenderer(
rasterizer=MeshRasterizer(
    cameras=torch_cameras,  # Camera parameters
    raster_settings=raster_settings
),
shader=SoftSilhouetteShader(blend_params=blend_params)  # Soft silhouette shader for rendering
)

# Parameters to optimize: Camera rotation (R) and translation (T)
params_to_optimize = [torch_cameras.R, torch_cameras.T]
optimizer = torch.optim.Adam(params_to_optimize, lr=5e-2)

# Optimization loop
num_steps = 200  # Number of optimization steps
for i in range(num_steps):
    optimizer.zero_grad()

    # Render the image from the current camera pose
    rendered_image = renderer(meshs, cameras=torch_cameras)[..., 3]  # Only the alpha channel (silhouette)

    plt.imsave("test_render.png", rendered_image.detach().squeeze().cpu().numpy())

    # Compute loss (mean squared error between rendered image and ground truth image)
    loss = F.mse_loss(rendered_image, image_gt)

    optimizer.zero_grad()
    loss.backward()
    optimizer.step()

How can I solve this issue and get PyTorch3D to properly optimize the pose? What might be causing the problem?