Lower all-to-all communication volume in fft transposes

[JHopeCollins]

Currently we transpose a complex array before/after doing the fft/ifft in the preconditioner.
The forward transpose before the fft and the backward transpose after the ifft could be on real arrays, which would halve the communication volume. This would need a second pencil/transfer to be created for the real transposes.

[JHopeCollins]

The mpi4py-fft module uses Alltoallw for the transpose, which can take different lengths and types of data on each rank. As far as I know this is rarely optimised by vendors because its so general, so is just implemented as a big isend/irecv round.
If we can change this to Alltoallv then it might improve the performance because this is more likely to be optimised for proper nlogn collective performance.

[JHopeCollins]

• Benchmark current implementation (all complex transposes and alltoallw)
• Benchmark reduced transpose volume (with alltoallw)
  • real-complex transposes
  • reduced precision
• Benchmark alltoallv (all complex transposes and alltoallv)
• Benchmark both modifications (lower communication volume and alltoallv)

[JHopeCollins]

We could also try communicating in lower precision. All of the computation is done with PETSc so we are limited to double precision there (or whatever PETSc has been compiled with), but the transposes are just mpi4py calls with numpy arrays. We could do these (and the fft/ifft) with a lower precision to reduce communication volume again.