Ok,
I downloaded the reference design and looked into the code and found some registry settings. I put those on the board and got a clock on pins D4 and D5 (not H4 and H5 as was written in the user manual). The clock frequency is not the right one since register 0x01F of AD9517 is sometimes E sometimes F. This means that PLL locks most of the time, but also that VCO, ref1 and ref2 are outside the frequency threshold. I measured the frequency with an oscilloscope and got 242.5MHz instead of 245.76MHz (value that the reference design says it should give).
I need to have a stable frequency at 245.76MHz, how do I get that and why do the base design settings don't give a stable clock at the right frequency?
Alex.
P.S. I looked at the vhdl and I have two comments. 1st, why is there no VHDL version, 2nd in the UCF file of the system there are no FPGA pins mapped to FMC pins H4 and H5 so please check the user guide pinout.