NEORV32 Multicore processor TMR implementation

[Luperior]

Good afternoon everyone,
for my Master Thesis I am working on a multicore version of the Neorv32 in order to reduce the error probability due to bit flipping in the bitstream. My idea is to triplicate the CPU (done already) and then introduce a Triple Modular Redundancy "security gate" before writing data into memory. Basically I want to compare the results obtained by the three different cores and then store the one which, in case of disagreement, is most obtained (eg 2 times out of 3).
In order to do so, I modified the neorv32_cpu_bus.vhd file by adding a comparison between the result coming from the same core ALU (wdata_i) and the ones coming from the other two cores ALUs (wdata_ix and wdata_iy). Should the three CPUs all produce different results, every single one will store the data obtained by their own ALU. This is the code:

mem_do_reg: process(rstn_i, clk_i)
  begin
    if (rstn_i = '0') then
      d_bus_wdata_o <= (others => '0');
      d_bus_ben_o   <= (others => '0');
    elsif rising_edge(clk_i) then
      if (ctrl_i.bus_mo_we = '1') then
	  d_bus_ben_o <= (others => '0'); -- default
	    -- Check for a majority vote (2 out of 3 match)
		if (wdata_i = wdata_ix) or (wdata_i = wdata_iy) then
        case ctrl_i.ir_funct3(1 downto 0) is
          when "00" => -- byte
            d_bus_wdata_o(07 downto 00) <= wdata_i(7 downto 0);
            d_bus_wdata_o(15 downto 08) <= wdata_i(7 downto 0);
            d_bus_wdata_o(23 downto 16) <= wdata_i(7 downto 0);
            d_bus_wdata_o(31 downto 24) <= wdata_i(7 downto 0);
            d_bus_ben_o(to_integer(unsigned(addr_i(1 downto 0)))) <= '1';
          when "01" => -- half-word
            d_bus_wdata_o(15 downto 00) <= wdata_i(15 downto 0);
            d_bus_wdata_o(31 downto 16) <= wdata_i(15 downto 0);
            if (addr_i(1) = '0') then
              d_bus_ben_o <= "0011"; -- low half-word
            else
              d_bus_ben_o <= "1100"; -- high half-word
            end if;
          when others => -- word
            d_bus_wdata_o <= wdata_i;
            d_bus_ben_o   <= "1111";
        end case;
        elsif wdata_ix = wdata_iy then
          case ctrl_i.ir_funct3(1 downto 0) is
          when "00" => -- byte
            d_bus_wdata_o(07 downto 00) <= wdata_ix(7 downto 0);
            d_bus_wdata_o(15 downto 08) <= wdata_ix(7 downto 0);
            d_bus_wdata_o(23 downto 16) <= wdata_ix(7 downto 0);
            d_bus_wdata_o(31 downto 24) <= wdata_ix(7 downto 0);
            d_bus_ben_o(to_integer(unsigned(addr_i(1 downto 0)))) <= '1';
          when "01" => -- half-word
            d_bus_wdata_o(15 downto 00) <= wdata_ix(15 downto 0);
            d_bus_wdata_o(31 downto 16) <= wdata_ix(15 downto 0);
            if (addr_i(1) = '0') then
              d_bus_ben_o <= "0011"; -- low half-word
            else
              d_bus_ben_o <= "1100"; -- high half-word
            end if;
          when others => -- word
            d_bus_wdata_o <= wdata_ix;
            d_bus_ben_o   <= "1111";
        end case;
        else
          -- No majority vote, there's a disagreement
        case ctrl_i.ir_funct3(1 downto 0) is
          when "00" => -- byte
            d_bus_wdata_o(07 downto 00) <= wdata_i(7 downto 0);
            d_bus_wdata_o(15 downto 08) <= wdata_i(7 downto 0);
            d_bus_wdata_o(23 downto 16) <= wdata_i(7 downto 0);
            d_bus_wdata_o(31 downto 24) <= wdata_i(7 downto 0);
            d_bus_ben_o(to_integer(unsigned(addr_i(1 downto 0)))) <= '1';
          when "01" => -- half-word
            d_bus_wdata_o(15 downto 00) <= wdata_i(15 downto 0);
            d_bus_wdata_o(31 downto 16) <= wdata_i(15 downto 0);
            if (addr_i(1) = '0') then
              d_bus_ben_o <= "0011"; -- low half-word
            else
              d_bus_ben_o <= "1100"; -- high half-word
            end if;
          when others => -- word
            d_bus_wdata_o <= wdata_i;
            d_bus_ben_o   <= "1111";
        end case;
        end if;
	  end if;
    end if;
  end process mem_do_reg;

Clearly, in order to make all of this possible, I added two extra IN signals for each CPU (wdata_ix and wdata_iy) and modified the "rs2" signal to be (IN)OUT since it represents the ALU outcome and has to be sent to the other cores. So for example the first core takes in cpu2.rs2 and cpu3.rs2 and outputs cpu1.rs2 and so on... however it does not work.
I am using Vivado to deploy it on a XILINX PYNQ Z2 FPGA and Synthesis and Implementation complete without any major warning (nor error obviously), and the Implemented Design makes total sense. I already tested the code and bistream end everything works fine without the TMR implementation so it shouldn't be a problem caused by the core triplication.
How is it possible then that I obtain nothing coming from the UART0 when I should see the first CPU output (after being "TMR-checked")?
Thank you all in advance!

Edit: I also tried other different approaches, like using Shared Variables or istantiating a new component which played the role of the TMR majority voter. However, let alone that neither of them worked, the first one is likely more subject to errors due to bit flipping and the second one (probably) had synchronisation issues since the values were available a few clocks after they were needed.
Should you have any other suggestions I am open to hear them, thanks!

[NikLeberg]

Hi there @Luperior super interesting thesis project! I wished I had such a thesis. :)
I'd not route the result of one cpu's ALU to the next cpu (the INOUT stuff). My professor highly discuraged us to ever use INOUT except in the toplevel.
You do the triplication on the cpu level, right? As yourself said, I'd add an additional component between the cpu's bus access to the intercon i.e. between the "Core Switch" and the "Gateway". Three cpu's issue read/writes and after majority voting a single correct read/write is actually issued. You won't check every result of every instruction but only the actual memory operations. You then basically have three cpu's independent that should always read or write the same data from/to the bus at all times. Whenever that is not true you can do a majority voting.
As linked by @stnolting my neorv32_soc SMP project is nowwhere near to be finished, but if I can explain or help in any way don't hesitate to ask.

[Luperior]

Thank you so much @NikLeberg for all your enthusiasm!
Could you explain a little bit more where exactly would you put the TMR module and how? What I mean is that, when I tried with a dedicated TMR entity, I chose to instantiate it in the neorv32_top file and it would collect the three "d_bus_wdata_o" signals coming from the three neorv32_cpu_bus entities and then output the TMR result, which was later sent to these buses so that the correct value would be stored into memory (just like you're saying I guess). However it did not work, and in my opinion it was due to the fact that signal assignments take place at the end of the clock time and thus I was introducing a delay of N (=2 in my case) T_clk: basically I had the result ready at time 0 but they were written at time 2, after all the TMR verification, when instead the CPU expected the result of instant 2 to be written. As a result, clearly, nothing worked despite Synthesis and Implementation were completed correctly.
So my questions are two:

1. How does your suggested approach differ from mine? Where could I put the TMR entity and how can I avoid adding delays?
2. My current approach should operate all within the same T_clk (it was my main concern): why doesn't it work? Can it all be due to the INOUT declaration?

Thank you again for your help!

[stnolting]

Hey @Luperior!

This really sounds like an interesting project! Here are some thought about this:

Do not use input ports for FPGA-internal logic - the synthesis tool might turn this into unexpected logic.

I agree with @NikLeberg. Do not change the CPU logic itself and implement the TMR system as part of the bus system (and not inside the CPU). This is where I would put the TMR module (red line):

All the buses of the three core complexes go into the TMR block. There, a majority logic is implemented. If the data to-be-written is identical across all three ports the memory access is actually forwarded to the further bus system (the DMA switch in this example). You can do that by gating the .we signal of the request bus. This should be possible without any further register stages (hence, no additional delay).

Btw, you are building a setup with three identical cores doing the same stuff at the same time. There is a (tiny?) chance that the synthesis tool is smart enough to identify this redundancy removing all but one cores.

[Luperior]

Do not use input ports for FPGA-internal logic - the synthesis tool might turn this into unexpected logic.

I'm guessing you mean INOUT right? Anyway thank you both, I'll change that.

All the buses of the three core complexes go into the TMR block. There, a majority logic is implemented. If the data to-be-written is identical across all three ports the memory access is actually forwarded to the further bus system (the DMA switch in this example). You can do that by gating the .we signal of the request bus. This should be possible without any further register stages (hence, no additional delay).

I think I get what you mean but I'm just not sure about why my current implementation doesn't work. I mean, I am not enabling and disabling memory writing with the .we signal but I am making sure that the stored value is always the most obtained one, thanks to a TMR logic implemented directly in the cpu_bus file (as shown in the first comment). Isn't it what you're saying by "Do not change the CPU logic itself and implement the TMR system as part of the bus system" ?

Could it all be due to the optimization you mentioned that may not instantiate "enough" components as maybe proven by the Issue (#663) ?

[stnolting]

I'm guessing you mean INOUT right? Anyway thank you both, I'll change that.

Yeah sorry, typo... 😉

Isn't it what you're saying by "Do not change the CPU logic itself and implement the TMR system as part of the bus system" ?

That is hard to say without looking at all of the code. The way you have implemented the TMR logic inside the CPU's bus unit will result in a write request issued by all of the three cores. How do you merge these three buses into the "final" (single) SoC bus? Several instances of the bus mux won't work here as they prioritized one of the two access ports.

That's why I am suggesting to do all the TMR stuff in a new CPU-external module (as part of the bus system). Then you have a single module where you can do the majority vote instead of distributing this across all the cores. Furthermore, such a single module attempt might improve timing as the signals to-be-checked do not have to travel there and back again. 😅