Memory Access

Memory Access on the RedPitaya

Memory Architecture

The RedPitaya uses shared memory for communication between the FPGA and the ARM processor. The memory is aligned to 4 bytes and has a width of 32 bits. This means that if you tell the processor to read address 0x40000000 it will read four bytes starting with address 0x40000000. Hence, it will read the addresses 0x40000000, 0x40000001, 0x40000002 and 0x40000003. Direct access to the unaligned addresses 0x4000000(1..3) is disallowed because no matter what address is given to the processor, it will always read four addresses starting with the one given. This would open up the possibility for a process to write and read from memory that is owned by another process which leads to huge security and stability risks.

Register Map

The RedPitaya developers created the following register map that is largely followed by pyrpl. The original RedPitaya map can be viewed here (https://redpitaya.readthedocs.io/en/latest/developerGuide/software/build/fpga/regset/2.00-18/list.html).

	Start	End	Module Name	Usage in pyrpl
CS[0]	0x40000000	0x400FFFFF	Housekeeping	Housekeeping
CS[1]	0x40100000	0x401FFFFF	Oscilloscope	Oscilloscope
CS[2]	0x40200000	0x402FFFFF	Arbitrary signal generator (ASG)	Arbitrary signal generator (ASG)
CS[3]	0x40300000	0x403FFFFF	PID controller	Dsp module (interconnects modules)
CS[4]	0x40400000	0x404FFFFF	Analog mixed signals (AMS)	Analog mixed signals (AMS)
CS[5]	0x40500000	0x405FFFFF	Daisy chain	FREE
CS[6]	0x40600000	0x406FFFFF	FREE	Was free, now used for FADS by me
CS[7]	0x40700000	0x407FFFFF	Power test	FREE

Memory access (FPGA side)

pyrpl is using its own simplified system bus architecture. Usually when using the Zynq FPGA series, the AXI bus is used to exchange data between different the processing system (the ARM processor, PS) and the programmable logic (the FPGA, PL). However, the AXI protocol is quite complex so the pyrpl developers opted to create their own system bus structure. To use the system bus, both a read and write interface have to be implemented and the module needs to be connected to the bus in the top level module. There, the base address (0x40?00000, ? replaced by 0..7 for the respective module) is also given to the module instance.

Top module level

pyrpl creates the eight big registers (0x40?00000, ? = 0..7) for the individual modules in a system bus chunk in the top module. This is also the location where the AXI bus is wired to the pyrpl system bus. To make use of the free registers, the lines have to be removed that set everything in the desired memory range to zero. Below, the range 0x40600000 - 0x406FFFFF is enabled to be used later for the FADS module.

// unused system bus slave ports

assign sys_rdata[5*32+:32] = 32'h0; 
assign sys_err  [5       ] =  1'b0;
assign sys_ack  [5       ] =  1'b1;

//assign sys_rdata[6*32+:32] = 32'h0;
//assign sys_err  [6       ] =  1'b0;
//assign sys_ack  [6       ] =  1'b1;

assign sys_rdata[7*32+:32] = 32'h0; 
assign sys_err  [7       ] =  1'b0;
assign sys_ack  [7       ] =  1'b1;

The special kind of indexing here is called indexed part select (more info https://stackoverflow.com/questions/17778418/what-is-and). Basically the sys_rdata wire is 8 times 32 bit wide and using this notation, the 32 bit words can be easily addressed.

To connect a module to the system bus in the top module, it must expose the following connections.

// Reset
.adc_rstn_i      (  adc_rstn                   ),

// System bus
.sys_addr        (  sys_addr                   ),  // address
.sys_wdata       (  sys_wdata                  ),  // write data
.sys_sel         (  sys_sel                    ),  // write byte select
.sys_wen         (  sys_wen[6]                 ),  // write enable
.sys_ren         (  sys_ren[6]                 ),  // read enable
.sys_rdata       (  sys_rdata[ 6*32+31: 6*32]  ),  // read data
.sys_err         (  sys_err[6]                 ),  // error indicator
.sys_ack         (  sys_ack[6]                 )   // acknowledge signal

To select the base address (0x40?00000, ? = 0..7) the respective number has to be used as index for sys_wen, sys_ren, sys_err, sys_ack and sys_cs. sys_rdata gets special treatment. The connected part of the 8 * 32 bit wide vector is the sixth 32 bit word.

Module level

To implement the system bus on the module level, the module needs to expose the system bus connections with the following set of inputs and outputs:

// Reset
input                 adc_rstn_i    ,  // ADC reset - active low

// System bus
input      [ 32-1: 0] sys_addr      ,  // bus address
input      [ 32-1: 0] sys_wdata     ,  // bus write data
input      [  4-1: 0] sys_sel       ,  // bus write byte select
input                 sys_wen       ,  // bus write enable
input                 sys_ren       ,  // bus read enable
output reg [ 32-1: 0] sys_rdata     ,  // bus read data
output reg            sys_err       ,  // bus error indicator
output reg            sys_ack          // bus acknowledge signal

Additionally, a sys_en wire has to be set up as follows in the module:

wire sys_en;
assign sys_en = sys_wen | sys_ren;

Reading from the system bus

To read values into registers from the system bus, the registers have to be defined in the module and need to be set to a default value upon the reset signal (in this case adc_rstn_i). The reset signal is active low, thus the adc_rstn_i == 1'b0 comparison.

// Registers for thresholds
// need to be signed for proper comparison with negative voltages
reg signed [DWT -1:0]droplet_threshold;
reg signed [DWT -1:0]sorting_threshold;

// Reading from system bus
always @(posedge adc_clk_i)
    // Necessary handling of reset signal
    if (adc_rstn_i == 1'b0) begin
        // resetting to default values
        sorting_threshold   <= 14'b00000000001111;
        high_threshold      <= 14'b00000011111111;
    end else if (sys_wen) begin
        if (sys_addr[19:0]==20'h00000)   sorting_threshold      <= sys_wdata[DWT-1:0];
        if (sys_addr[19:0]==20'h00004)   high_threshold         <= sys_wdata[DWT-1:0];
    end

Addressing

The implementation uses a compound way to assign addresses. On the top module, the base address (the 12 most significant bits) is assigned to a module using the sys_wen, sys_ren, sys_rdata, sys_err, sys_ack and sys_cs wires, while on the module level, the remaining 20 bits of the address are handled.

In the code example, two addresses are used: 0x40600000 and 0x40600004 for the two registers sorting_threshold and high_threshold. To use different addresses, the least significant 20 bits have to be changed but can only be multiples of 4 because of the 4 bit alignment.

Writing to the system bus

Writing values to memory works similar to reading from memory. First all registers that should be written to memory need to exist and they need to be initialized somewhere in the module (done in the reading section for the thresholds). Upon the acive low reset signal adc_rstn_i the sys_ack has to be set to zero. In any case the sys_err has to be set to zero.

// Registers for droplet counters;
reg [MEM -1:0]droplets = 32'd0;

// Writing to system bus
always @(posedge adc_clk_i)
    // Necessary handling of reset signal
    if (adc_rstn_i == 1'b0) begin
        sys_err <= 1'b0;
        sys_ack <= 1'b0;
    end else begin
        sys_err <= 1'b0;
        casez (sys_addr[19:0])
        //   Address  |       handling bus signals        | creating 32 bit wide word containing the data
            20'h00000: begin sys_ack <= sys_en;  sys_rdata <= {{32- DWT{1'b0}}, sorting_threshold}  ; end
            20'h00004: begin sys_ack <= sys_en;  sys_rdata <= {{32- DWT{1'b0}}, high_threshold}     ; end
            20'h00008: begin sys_ack <= sys_en;  sys_rdata <= {{32- MEM{1'b0}}, droplets}           ; end
            default:   begin sys_ack <= sys_en;  sys_rdata <= 32'h0                                 ; end
        endcase
    end

Full 32 bit words are required for writing to memory. Therefore, registers that are not 32 bit wide like the 14 bit wide thresholds have to be padded with leading zeros. This is done by concatenating 32 - register width zeros with the register value ({{32- DWT{1'b0}}, high_threshold} DWT being 14).

Addressing

Addressing works the same way as reading from memory, but the structure to handle the different addresses is now a casez instead of multiple if statements. I have not checked if this makes a big difference, however, it is implemented like this all over pyrpl and the original RedPitaya code. The addresses have to be the same for reading and writing to memory for a single register.

Memory access (client side)

Accessing the addresses from the python module is easy compared to the PL. The memory access is handled in the hardware_modules subpackage of pyrpl.

For each class that inherits from HardwareModule, the addr_base class attribute sets the base address of the module and individual registers can be mapped using the classes defined in the attributes module. In this case the FloatRegister class is used to map the 14 bit thresholds to the class attributes.

from ..attributes import *
from ..modules import HardwareModule
from ..widgets.module_widgets.FadsWidget import FadsWidget


class FADS(HardwareModule):
    """
    Implementing fluorescence activated droplet sorting
    """
    addr_base = 0x40600000
    name = 'FADS'
    _widget_class = FadsWidget

    _gui_attributes = ["low_threshold",
                       "high_threshold"]

    _setup_attributes = _gui_attributes

    _adc_bits = 14

    low_threshold = FloatRegister(0x0, bits=_adc_bits, norm=2 ** 13 / 20,
                                  doc="low threshold for sorting")
    high_threshold = FloatRegister(0x4, bits=_adc_bits, norm=2 ** 13 / 20,
                                   doc="low threshold for sorting")

More information about the pyrpl architecture can be found in the GUI Architecture wiki.