Topic: PC720 PCIe Communication Unresponsive  (Read 11216 times)

Benjamin.Andrews October 08, 2013, 12:46 PM

  • Member
  • *
  • Posts: 11
I am working on a PC720 FPGA card, and am having problems communicating over PCIe with my custom design.  When I use the reference design provided with the card there is no issue with PCIe communication.  In my simulation, I am able to setup a PCIe link and send data across with no problems.  Is there a way to get a debug version of your API to verify that a link is being setup correctly and that it is just the data that not being send across?  I am fairly certain that the FPGA is enumerating within the PCI requirement, as it is displaying as expected in Device Manager and allocating the desired amount of memory.
Regards,
Ben Andrews

arnaudNL October 09, 2013, 03:35 AM (#1)

  • 4DSP Staff (EU)
  • Administrator
  • Member
  • *****
  • Posts: 7110
Dear Ben,

We do not have debug builds because it is generally not required, its been year we haven't step traced this code. We know it works so the problems are generally somewhere else (Firmware or high level application). You should know that the command bus (ie read/write register) is done via PCI BAR 0 so if the device is properly enumerated it should be fine (You can check with PCItree (free software) whether or not the card is properly mapped, try to read first words of PCI BAR 0 and it should not read 0xFFFFFFFF). The data transfers are  using DMA, the DMA engine is configured via BAR also and the data is transferred by our FPGA.

Note that our Linux BSP is delivered with full source code (API and device driver) but is a sold option.

It could be handy if you could describe what does not work!

Best Regards,
Arnaud

Benjamin.Andrews October 09, 2013, 06:02 PM (#2)

  • Member
  • *
  • Posts: 11
Arnaud,
I generated the reference design .mcs file from the extracted firmware in the 308_pc720_fmc176 folder.  I then loaded the flash following the instructions in the user's manual and power cycled my computer.  The card came up correctly, but there is now no 4DSP device being displayed in my device manager.  I also tried opening the 4DSP GUI and it was unable to find the card as well.  I have tried powercycling my PC to confirm the reference design is being loaded.  I have also used Impact to verify the .msc file was written into the flash correctly.  I am unsure why the PCIe would not work with the card with the reference design loaded.  I have tried another 4DSP card with the default firmware loaded, and it is recognized correctly.  Is there a step I am missing in recreating the reference design?  Could you send me a .bit or .msc file to load on to the card, in order to determine if it something in building the design? 
Regards,
Ben Andrews

ebarhorst October 10, 2013, 07:47 AM (#3)

  • 4DSP Staff (EU)
  • Administrator
  • Member
  • *****
  • Posts: 1222
Hi Benjamin,




if you do a restart of your computer is the card then detected?


if not;
in order to receover the flash contents can you try the following for me:
1)load  the FPGA with the bit file found in
C:\Program Files (x86)\4dsp\Common\Firmware\Recovery\308_pc720_fmc176
2) restart your computer (not a power cycle)
3) check if the board is detected and select it from the 4FMGUI
4) in the registers/update tab for teh fpga a firmware path select the hex file found in
C:\Program Files (x86)\4dsp\Common\Firmware\Recovery\308_pc720_fmc176
5)press the load button
6) wait until programmed successfull appears
7) turn off the computer
8)turn on the computer
9) check if the board is detected in windows.




if that works can you try the same steps(starting at step3 since the board should load correct from flash again) with a .hex file generated from you own bit file. You can use the bit to hex tool for that. This tool is located
C:\Program Files (x86)\4dsp\4FM Core Development Kit\Bins
just drop the bit file onto the application.




can you let me know if that works?




thanks,
Erik

Benjamin.Andrews October 10, 2013, 10:12 AM (#4)

  • Member
  • *
  • Posts: 11
Erik,
Thank you for your help! The board is working with the reference design now.  The only minor issue I had was that the board did not get recognized after power cycling the computer, but when I did a reset it get did get recognized.  I think there is a enumeration timing problem there, but the work around is just to restart the computer. 
 
Does the 4FMGUI use the PCIe connection to write the hex file onto the board?  And is that functionality included in the API that was delivered with the BSP? 
 
Now that I am confindent there are no problems with the card I will go back to debugging what is wrong with my PCIe setup with my custom FW load. 

arnaudNL October 10, 2013, 11:48 AM (#5)

  • 4DSP Staff (EU)
  • Administrator
  • Member
  • *****
  • Posts: 7110
Dear Benjamin,


Yes, the GUI is using PCIe for the upload of firmware, to be more precise the data is transferred between the host and the pc720 by the DMA engine implemented in the PC720 firmware.


Yes, the function to upload to flash is part of the API, _4FM_UploadFirmware(). This is what the GUI uses in order to program the flash. You can find some information in the 4FM Programmers Guide.


I am also attaching the source file used to compile the tools found in the bins folder. The documentation is not clear around what is supported on PC720, only FLASH to normal image is supported. The upload goes on to FPGA A, when fpga_dest is set to 'FLASH_A' in uplfw.c.


The other argument, mode is actually discarded you can use, 'CREG_BASED' as per uplfw.c


I hope that helps,
Arnaud

Benjamin.Andrews October 14, 2013, 09:25 AM (#6)

  • Member
  • *
  • Posts: 11
Arnaud,
Going back to your response from Oct 9th. 
 
Quote
We do not have debug builds because it is generally not required, its been year we haven't step traced this code. We know it works so the problems are generally somewhere else (Firmware or high level application). You should know that the command bus (ie read/write register) is done via PCI BAR 0 so if the device is properly enumerated it should be fine (You can check with PCItree (free software) whether or not the card is properly mapped, try to read first words of PCI BAR 0 and it should not read 0xFFFFFFFF). The data transfers are  using DMA, the DMA engine is configured via BAR also and the data is transferred by our FPGA.

Using a VIO through the JTAG port I am able to read and write registers inside the FPGA.  The BAR0 register is set to xFC00_0000, which corresponds to the address that I see in the device manager on the PC.  Furthermore, when I look at the link status register I see that the link is up.   However when I go to write to the FPGA using the API function "_4FM_WriteRegister", I get the _4FM_OK response but when I look at the reigster through the VIO it did not write properly.  And if I attempt to read the register with the API function "_4FM_ReadRegister" I get an error that I couldn't read the register.  Is there some additional PCIe/FPGA setup that I am missing?

Thanks,
Ben 
 


 

arnaudNL October 14, 2013, 10:31 AM (#7)

  • 4DSP Staff (EU)
  • Administrator
  • Member
  • *****
  • Posts: 7110


Ben,


_4FM_WriteRegister is writing to the BAR directly. _4FM_ReadRegister is reading from the BAR directly, this cannot fail. It is translated to writing to PCI addresses. Maybe you are using wrong addresses. I think the BAR is 64kB wide, and an address is word address (Register 0 is byte address 0 to 3, Register 1 is byte address 4 to 7, etc..). This cannot fail normally. Typically try to use _4FM_WriteRegister with an address of 0 and value of 1, this is what _4FM_Reset does, writing a '1' at address 0, which is the 4 first bytes of the BAR.


The debug version of the API will not help you; there is a simple IOCTL call we use for years already, this is sent to a device driver which is in charge of sending that to the correct hardware address.


Here is the source code for the read register function:


Code: [Select]
_4FM_error_t _4FM_CALL _4FM_ReadRegister(_4FM_DeviceContext *ctx, unsigned regno, unsigned int *value)
{
  if (!ctx)
    return FDSP_ERROR1(ERR_NO_CTX, MSG_NO_CTX);


  if (_4FM_IoCtl_WrRd(ctx, IOCTL_4FM_RD_REG, &regno, sizeof(regno), value, sizeof *value))
    return FDSP_ERROR1(ERR_IO_FAIL, MSG_IO_FAIL);


  return FDSP_OK;
}


Here the most relevant IOCTL code from the device driver, besides deep system calls, this is the place that could fail if the address is bigger than 16k (16k * 4 = 64k). It is pretty weird, the same code is there on the Write side. Note that this code translates to a READ_REGISTER_ULONG() call to the address of bar+addr*4.


Code: [Select]
if( RegNo * sizeof(ULONG) >= 64*1024 ) {
              status = STATUS_INVALID_PARAMETER;
              TraceEvents(TRACE_LEVEL_INFORMATION, DBG_PNP,
                  "Register not withing current BAR: %lu", RegNo);
              break;






Is this happening with a the PC720 firmware when recompiled and not modified? We have 10 engineers working on PC720 and they all been doing release based on our PC720 host interface. It sounds like you did not reuse our material as-is but you are modifying/rewriting instead.


Also keep in mind updating PC720 FPGA when Windows is running require to reboot Windows as the PCI engine is in the FPGA. Typically Windows still knows/found the device so you could be communicating with a ghost. This is easily spotted out, reading from any address of the BAR will return -1 (0xFFFFFFFF) in this case. PCI tree is perfectly able to read/write from any PCI devices in the system.


As explained,  first four bytes of the BAR is the reset register, write a '1' and see if you get any LED activity or triggers in your design. Then try to read some known addresses. Address 0x11 and 0x12 should return so diagnostics information. If you extended the BAR space and try to read/write above 64kB, then you know the driver will not allow you to do that. Once again PCITree is handy here for these simple read/write to BAR. There is also a commercial solution, called windriver from jungo, there are evaluation version available, it will allow you to code your own API wrapping around a generic PCI driver.


Best Regards,
Arnaud























[/code]

Benjamin.Andrews October 14, 2013, 11:03 AM (#8)

  • Member
  • *
  • Posts: 11
Arnaud,
Thank you for the very detailed reply!  I will dig into more throughly and give you a detailed response, but one thing sticks out right away.  For my custom FW design I am allocating 16M of memory space for the card, and it seems from you post that 16K is the maximum amount that the driver is able to handle.  Would there be some way that we could work with you to update the driver, or would we have to create our own driver to handle that large of a memory space?
 
Thanks,
Ben

arnaudNL October 14, 2013, 11:22 AM (#9)

  • 4DSP Staff (EU)
  • Administrator
  • Member
  • *****
  • Posts: 7110
Dear Ben,


Yes, the driver knows the device has 64kB of BAR so will not allow any write beyond this range as it could cause writing to other device or worse, to kernel memory. PCITree would be the best player here because he will allow you to write/read the whole BAR, at least for testing your implementation


The device driver will need to be recompiled in order to remove this limitation, I guess the driver should not have the hard coded limits but work with the BAR size instead. Re implementing that could make sense but it will need to go through the sales channel in order to have someone allocated on this task.


Thanks,
Arnaud

Benjamin.Andrews October 14, 2013, 01:05 PM (#10)

  • Member
  • *
  • Posts: 11
Arnaud,
Can we talk to the sales department today to get started on updating the driver?  We have been working through Micheal Brown for ordering the boards, but there may be another contact in the sales department for this type of work.  If we could get on a call this afternoon that would be excellent.
 
Thanks,
Ben

arnaudNL October 14, 2013, 05:35 PM (#11)

  • 4DSP Staff (EU)
  • Administrator
  • Member
  • *****
  • Posts: 7110
Dear Ben,


Yeah Michael is the right person to contact. He is in close contact with all the engineering chiefs.


Best Regards,
Arnaud

arnaudNL April 01, 2014, 08:03 AM (#12)

  • 4DSP Staff (EU)
  • Administrator
  • Member
  • *****
  • Posts: 7110
This topic is being closed because the issue is considered as resolved by 4DSP. Feel free to create a new topic for any further inquiries.