[Cryptech Core] Fwd: FPGA issues (Was: Re: Current sync problem)

[Joachim Strömbergson]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Aloha!

(As Randy correctly pointed out, this should be on core or even on @tech.)

- -------- Original Message --------
Subject: FPGA issues (Was: Re: Current sync problem)
Date: Fri, 19 Dec 2014 14:57:01 +0100
From: Joachim Strömbergson <joachim at secworks.se>
Reply-To: joachim at secworks.se
To: Basil Dolmatov <dol at reedcat.net>
CC: Randy Bush <randy at psg.com>, Paul Selkirk <paul at psgd.org>,  Rob
Austein <sra at hactrn.net>, Fredrik Thulin <fredrik at thulin.net>, Leif
Johansson <leifj at sunet.se>

Aloha!


Basil Dolmatov wrote:
> Can you provide some description of current timing problems (maybe 
> other open issues ) with existing FPGA design?

Sure. We need to separate the two though.

The timing problems are related to how the EIM bus from the Freescale
CPU works, the board design of the Novena, the timing setup (the timing
constraints) by Bunny for the FPGA and finally the clock scheme used in
the FPGA by Bunnie.

If we go backwards, there are several clocks generated at the top level
of the FPGA by Bunnies design. One problem we have is that these clocks,
even if they are even divisors of each other (66MHz being the 133 MHz
clock divided by two), the actual HW-block allocated by Bunnie means
that the clock flanks are not aligned, but there is a slight difference
between them. Also, due to the HW-blocks used, the clock trees are
possibly not allocated in the correct way (this is something I'm
investigating). This means that the clocks might not be distributed over
the chip on an even way as a balanced clock tree provides.

To further complicate matters, there are more than one external clock
feeding the design and they are quite probably not correlated. The 133
MHz clock is also not always available (it becomes active during first
operation on the EIM.)

If we move back up, the EIM seems to be a fairly finky bus (as described
by the post Rob found) which means that it has tight timings and unless
they aren't met (valid data isn't stable at the flank), errors will
happen. To complicate matters, the bus moving over the PCB will take non
zero time. Also capacitance between wires will smooth out the flanks.
This is not unique for the Novena, but since the timing for the EIM is
tight, these effects eats away at the small margin. The proper way for
handling this is to add constraints in for the FPGA tool to allow it to
know about the timing requirements on the EIM. There are constraints in
the UCF-file for the Novena provided by Bunnie. The problem we have is
to easily verify of they are correct or not.

What we can see is that ISE complains about excessive hold time
violations when building a design. And these complaints are not only for
our designs, but even if you build a design without any cores, but
simply a few test registers. Excessive hold violations means that the
data will not be stable on the flank and may thus be sampled
incorrectly. And the incorrect behaviour may vary with the actual data
(signals wider that a single bit may effect each other), and also due to
slight differences between placement, routing on the chip. The net
effect is that it is hard to get a stable design. And that rebuilding a
design without actually changing anything in the design may cause
problems to appear in different parts of the design and with different
behaviour. Very hard to debug.

Our cores cannot be clocked at 133 MHz, they top out at something like
100-80 MHz in the given FPGA family. It is not really a specific problem
with our cores, but fairly reasonable for cores of this type. One could
work on making them clocked at higher speed, but that would require a
lot of pipelining probably at least doubling (or more) the number of
cycles a given operation takes.

This in turn means that our cores needs to be clocked using another
clock. 66 MHz should be a good choice that provides good timing margin
in the cores.

In order to communicate with the CPU over the EIM we then need to cross
over the clk133 to clk66 boundary to get data and commands into the
cores. And then back from clk66 to clk133 again. If the two clocks are
guaranteed to be stable it _should_ be easy to do this. Just ensure that
anything asserted with clk133 is valid for at least two cycles and it
should have been captured correctly in the clk66 domain. The opposite is
easier since anything asserted in clk66 domain will be asserted for at
least one clk133 cycle.

But if the clock are in fact not aligned or they aren't even in sync
with eachother, hold issues will arise, as well as meta stability issues
etc. Basically we would need a dual clocked FIFO in each direction to
help us traverse the boundary. And even then MTBF issues will occur.

What I have been doing is first to just draw a diagram of the clock
system designed by Bunnie too see if we can rebuild it using propoer
clock generator macros that do provide clocks that are in sync with
eachother and that also are guaranteed to be connected to the balanced
clock trees in the FPGA. This is not completed yet.

I've also started to consider adding a specific tight register-register
module that can be placed at the boundary and ensure that commands and
responses are sampled with the given clocks wit as short delay path as
possible. Something that sits in front of coretest and stretches all
accesses to at least two cycles. This _should_ be possible to do without
actually updating the cores.

What we do know is that the cores themselves do work running at 66 MHz
when used by the I2C interface so it is not the cores at the given clock
frequency that is the problem. At least this is my understanding of the
problem.


To answer your second question, there are quite a few things that needs
to be developed for the cores. Esp the TRNG still lacks the debug
functionality, the pseudo entropy source including access mux to the
CSPRNG output, AIS31-based on-line tests of entropy providers. Just
moving the TRNG from version 0.1 closer to 1.0. Then we have things like
ESSIV mode for doing key wrapping etc.


> I am currently sorting out FPGA developers to find one (or couple)
> to attach to the project and this can be quite good test case to
> find most suitable person for us.

What would really help us forward esp with the EIM issue is somebody
that is really good at clocking, specifically in Xilinx FPGAs. Somebody
that could work out the correct clocking implementation including
getting a good clk66. And then aid us in how to do the boundary crossing.

I believe that person would also need to understand board timing to be
able to verify and quite possibly correct the eim constraints to remove
the hold time violations. This would not only help us, but the Novena
project as well as others that tries to use the FPGA on the Novena using
the EIM.


I hope this was fairly readable. If not, just ask.


- -- 
Med vänlig hälsning, Yours

Joachim Strömbergson - Alltid i harmonisk svängning.
========================================================================
 Joachim Strömbergson          Secworks AB          joachim at secworks.se
========================================================================
-----BEGIN PGP SIGNATURE-----
Version: GnuPG/MacGPG2 v2
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQIcBAEBCAAGBQJUlEjMAAoJEF3cfFQkIuyN4Y8QANNuIZ5eLQKTM+JVLbWBTZdg
4eyyN+y9Ex+86KczsaoVvgrFWA4l7THyC0dhoK1Yai8etJknWjgZFK7Axd6VntMB
rY/b/wrA/TFsO6W6eRVf5o3Z9g1+zjRfx7G3xSmViOJf2DQEdimwQZvSr2qY/lKq
eR1EKZsPHAiZWzNXMr1n6iIG1PL6SNxQOXpsJbG7q/7nKBtN9oMFN2Qd6he6f4Of
ladMWvJyAPOr6PWjjQhTiaaggW7pyX5qUhEIayBEhmrXP9ljW+w0ghA/SbBF25BI
fN1HXyKuEuTqloPgHQGJ/pcUy9fDOecJ1q032aQcjHMgBhvNHNlMqJNlc2M3v61F
I6cykklhQ6pXLpxrPZYevx1n1XDNkppL+BeulbXahRY1iCu/2yHiIFMBXe4dyNGW
NxOWc0Wn17IERW1ar7/j9I6tLoi3vwC9nB+r1xC3/37ipI8eZ4A5KC11rb8AU6zo
5xixhamyhWxTxgxXYkrnh0BCR5Fo2dvDiy4h1Nm06iONo0gCN+pOXxMXOAKjrWng
JKkTeWy5+C97oQ7za1IACL5vvg1EeHj45BrrZenN0/KwPtR328VOIXPVXq37+m3U
/0SwISOk9j4RNwyFCJKW8sST9bdl4cETQ2mgVpiPGP71jnVI40oOr9L4NpgBmyfP
rljZTZ2+FkEM8dVuw+kB
=atTE
-----END PGP SIGNATURE-----