Topic: FMC116 Initialization Problem  (Read 16506 times)

vicmarher July 25, 2014, 11:32 AM

  • Member
  • *
  • Posts: 43

Hi again.
I've used your FMC116 Reference Design as a base for my design.
I've replaced your SIP_MAC_ENGINE with a custom star accessed through an embedded app running on a LEON3 Soft-Core in my FPGA.
Anyway, the initialization source code is exactly the same as in your Reference Design "C" code, changing the functions that send the SIP Commands through ethernet, by functions that access I/O Registers with the Soft-Core, and re-route SIP_COMMANDS to ADC.


I've also updated fmc116 vhdl to avoid bursts and work in a continuos mode, using a double buffer that generates an IRQ when one of the two buffers is full. The App running in Soft-Core catches the IRQ and reads the content of the buffer.


The buffer in the output of each channel has also been changed from a 16 to 64 buffer to a 16x16 buffer with different clocks for read and write, which samples go directly to the just described double buffer. This buffer is used for the clock-domain change between ADC clock and Soft-Core clock.


When I power on FPGA and test a sinus signal in the FMC116, I read it correctly.
But when I interrupt (ctrl+C) execution and start it again withow powering off/on my FPGA (Xilinx Virtex ML605), the results are really strange, as you can see in the attached picture.


I've also created a simple sequence generator and plug it just before the 16x16 buffer. This way, I replace the ADC output by the sequence output., but the data path is still the same. The sequence can be enabled in real time with a switch in the FPGA Board. When I do that, the sequence is correctly read, but the sinus is still wrong read, showing in my opinion that my problem is in ADC initialization.


Attached figure shows the effects.
So the questions are :


1.- Is there any initialization needed for the FMC116 that is not taken into account in the Reference Software?


2.- The VCO calibration from clock-tree is treated diferently from power up and for subsequent executions (page 40 of clock-tree pdf AD9517-3). I've tried to reset 0x18 register and update registers with 0x232 reg before updating registers, but the behaviour is the same. How exactly must be treated VCO calibration for not the first time after power up? may it be my problem?


3.- I use the clock from LEON Soft-Core as master clock to be distributed for clock and command clock. It's a 125Mhz single-ended clock generated by a LEON PLL using the  200Mhz differential clock from FPGA. Is there any difference between this clock and the one that generates SIP_MAC_EMGINE in Reference Software?


I hope you can help me.


Thanks a lot!


Víctor Martín, IEEC

arnaudNL July 28, 2014, 04:16 AM (#1)

  • 4DSP Staff (EU)
  • Administrator
  • Member
  • *****
  • Posts: 7110
Dear Victor,


1. Not as far as I know, no
2. I think the problem is not there, we can run our reference design several times.
3. The stellarIP clock is indeed generally 125MHz.


I would check the following:


1. Data stuck in a queue somewhere? Your ctrl-c is going to interrupt a process and there for you should flush all the FIFOs. In the ETHAPI we do the follow sequence:


- Write twice to address 0, we write a 1 there. This causes the mac engine to reset the FPGA fabric. The software FIFO part of NDIS protocol are automatically flushed by the OS.
- Send the host's mac address.


2. Its quite a work around but, generally you would not want to interrupt the process, you would intercept the ctrl-c event, prepare the system to bail out (ie wait for data transfer to finish, etc..) and then finally exit. Check the following, http://stackoverflow.com/questions/4217037/catch-ctrl-c-in-c


I hope that helps,
Arnaud



vicmarher July 28, 2014, 05:30 AM (#2)

  • Member
  • *
  • Posts: 43
Hi Arnaud,
thanks for your reply.


I will try what you say, although I already do something similar. In fact, at startup, I disable all FMC116 channels, force all the buffers to be empty by setting read_enable to '1' untill they are empty, and only then I re-enable the channels and set IRQ handlers to read my buffers.


And in fact, even if buffers weren't empty in a second execution after ctrl+c, what i'd expect to receive would be samples from previous execution, in a perfect sinus wave, not an strange and noisy signal as the one you could watch in the attached figure.


Thanks!


Víctor

arnaudNL August 01, 2014, 06:48 AM (#3)

  • 4DSP Staff (EU)
  • Administrator
  • Member
  • *****
  • Posts: 7110
Dear Victor,


I was following up on topic inactive for a couple of days.


Any breakthrough on your side?


Thanks,
Arnaud

vicmarher August 01, 2014, 07:27 AM (#4)

  • Member
  • *
  • Posts: 43
Hi Arnaud,
unlucky this week it has been impossible for me to keep working on that.
But I insist in the fact that the clean up of the buffers is already done at the beggining, and in case it wasn't done, the results would have samples from the previous executions already in the buffers, so the plot would show a sinus that suddenly restarts in a new phase, not the strange noise image that you can see in the pictures.


Anyway, as soon as i can i will try capturing SIGINT and call the cleanup functions I call at the beginning, and I will tell you the results. Thanks for your interest! and off course, if you or anybody has any idea,  please let me know.




Many thanks!


Víctor

arnaudNL August 01, 2014, 07:38 AM (#5)

  • 4DSP Staff (EU)
  • Administrator
  • Member
  • *****
  • Posts: 7110
Hello Victor,


I am forwarding the topic to someone else, I looked again the buffer and indeed you might be right about the ADC init.


Still there must be a difference in your design as we can on our side run the designs several time without to power cycle the carrier.


Thanks,
Arnaud

vicmarher August 01, 2014, 07:52 AM (#6)

  • Member
  • *
  • Posts: 43
Hi again.
Off course the problem is in my side, there is no doubt about that.
I've modified things from your original design, but after several comparations file by file, I haven't found which is the difference that makes this error to happen.
Maybe is not a problem of initialization, maybe is a problem of syncing of any of the clocks, and my poor knowledge of fpga (this is my first project, and is not an easy one), makes it even more difficult to solve.


That's why i tried the sequence i showed you in the plots, to check if the problem was in the new data path that I had done changing the original design, or previous to that, in the ADC initialization/synchronization. And watching the plots I sent you, makes me think that my mistake is on that first stage of the execution.


I will try to check all the changes again, and send you a detailed list of the changes from your original design if you think it could be a good way to find the incorrect change.


Once again, many thanks!!!


Víctor, IEEC





iklink August 01, 2014, 11:47 AM (#7)

  • 4DSP Staff (EU)
  • Member
  • *
  • Posts: 153
Hi Victor,


Do you have the possibility to print messages the same way we do in our reference software? Maybe it is possible for you to get the output of the Training procedure of the ADC init, refer to last line of code of the FMC116_adc.cpp file. Maybe something goes wrong in the serdes bit alignment and the samples are corrupted, the erroneous pattern still looks like some kind of pattern... If you're missing or inserting data samples somewhere down the datapath I would expect the pattern would look different.


Best regards,
Ingmar van Klink

vicmarher August 01, 2014, 12:14 PM (#8)

  • Member
  • *
  • Posts: 43
Hi Ingmar,
thanks for your reply.
All the prints that were present on your reference design are still untouched in my project, and I can see them, so yes, I can pass you the output of the training bits. Unlucky today it will be impossible because I'm not at work now, but monday will be the first thing I will do.


I must recognize that I have no idea of what the serdes bit alignment is, and also the Training procedure.  I will try to read the documentation about it again and redo a check to see if I have changed anything about it in the code.


Many thanks!! Let's hope you have found the problem!!!


Best regards,
Víctor Martín

arnaudNL August 04, 2014, 03:50 AM (#9)

  • 4DSP Staff (EU)
  • Administrator
  • Member
  • *****
  • Posts: 7110
Hello Victor,


Please keep us updated when you have something, I keep this topic open for the time being!


Best Regards,
Arnaud

vicmarher August 04, 2014, 04:51 AM (#10)

  • Member
  • *
  • Posts: 43
Hi again!!
I've been running several times to try to catch a difference in the training bits and trying to find a pattern that were different when things where ok and when not.
I haven't found it...
I attach 6 logs, three of them when ok and the rest when KO. I hope (even beg :D ) you can find something.


On the other hand, there is something that has always been a serious doubt for me. The 4 frequencies of the 4 ADC chips, quite often appear to be different. (I know that I could use only one of them If I use only one channel, but ...)




Thanks again!


Víctor Martín

iklink August 04, 2014, 06:27 AM (#11)

  • 4DSP Staff (EU)
  • Member
  • *
  • Posts: 153
Hi Victor,


If think your data gets corrupt due to the input delay settings. There is a bit align machine inside the serdes interface in the FPGA. The Training procedure starts the bit align machine in the firmware. The bit align machine adjusts the FPGA input delays to be as far away from an edge of the LVDS data interface in order to prevent data from being corrupted depending on timing and temperature variations. The maximum span of the input delay adjustment is something like 2.5ns. I noticed that your software configures the ADC to sample at 10MHz. With a sampling period of 100 ns, the serialization interface from the ADC to FPGA runs at 160MHz having a period of 6.25ns. When the bit alignment machine tries to search for an edge in the digital data, but with a period of 6.25 ns and an maximum adjustment range of the input delay of the FPGA, the bit alig machine is not able to detect an edge and it will fail its training. This is the reason that you get the message Training Busy in your console where the message should actually be Training Ready.
There are two things you can try to tackle the problem.
First set the ADC sampling clock to 125MHz, and your data should be fine then.
Second, probably your application requires 10MHz. In order to use this, disable the automated Training procedure in the software. If the data is still erroneous, you will have to adjust the input delays of the FPGA manually untill your data is correct (this can be a on time effort to figure out). Please refer to the star documentation of the SIP_FMC116 star how to access the input delay registers.


Best regards,
Ingmar

vicmarher August 04, 2014, 06:45 AM (#12)

  • Member
  • *
  • Posts: 43
Hi Ingmar,
Many Thanks for your reply!




I need ADC to acquire at 10Mhz or even less. But I suppose that setting the ADC to 125Mhz and use a filter with high downsampling would be the same.


I will try what you say and keep you updated, let's hope that with good news!!

THANKS AGAIN!

Víctor Martín, IEEC



vicmarher August 04, 2014, 07:58 AM (#13)

  • Member
  • *
  • Posts: 43
Hi Ingmar,
I've restored the clock registers to keep ADC at 125Mhz, Training Status is Ready, and after more then 12 executions, all the output keeps being OK, so it seems that you have found the solution.


Many many many many thanks!!!!


Víctor

arnaudNL August 04, 2014, 08:38 AM (#14)

  • 4DSP Staff (EU)
  • Administrator
  • Member
  • *****
  • Posts: 7110
Perfect Victor and thanks Ingmar!


I am closing this topic for the time being, feel free to open topics any time you need to do so.


Best Regards,
Arnaud