Dear Sir,
This have nothing to do with the hardware yet; The software is not able to communicate with API or the Driver.
Try to run the software from C:\Program Files (x86)\4dsp\FMC Analyzer that should work better.
As far as the custom board is concerned, we do not provide support for custom boards unfortunately. If you keep the firmware same, the software should work as is. Note that an application FMC150App is provided with the development kit. The FMC Analyzer product is using the same code.
VC707, KC705 and ML605 are Xilinx development board officially supported, nothing else!
Best Regards,
Arnaud