[Open Crypto Project] #63: RPC does not recover gracefully from lost synchronization

[Open Crypto Project]

#63: RPC does not recover gracefully from lost synchronization
--------------------+----------------------------------------
 Reporter:  sra     |       Owner:
     Type:  defect  |      Status:  new
 Priority:  major   |   Milestone:  Alpha board DNSSEC signer
Component:  HAL     |     Version:
 Keywords:          |  Blocked By:
 Blocking:          |
--------------------+----------------------------------------
 If RPC client and server are not in lock step, we do not recover
 gracefully.  Eg, if one sends non-RPC bytes down the serial port before
 hooking up client and server (eg, because one is probing the serial port
 to find out which one it is), client and server get confused and do not
 recover without help.

 We're using SLIP framing, which should bracket any bogus bytes as an
 invalid packet, so client should just be able to check the opcode in the
 response to see if it matches what it was expecting, and, if not, discard
 the packet.

 Server (HSM) side seems to get confused too, though, haven't been through
 that with gdb yet to figure out exactly what's failing.  In experiments to
 date I've had to use the console to reboot the HSM to recover from this.

 Sample failed exchange, after using `cryptech_probe` to identify ports.
 Client sends:

 {{{
 passing to serial port:
 00 00 00 03 00 00 00 05
 00 00 00 02 00 00 00 05
 66 6e 6f 72 64 00 00 00
 }}}

 Client receives:

 {{{
 serial port received response:
 40 00 44 00 20 01 36 3c
 00 00 00 1d
 }}}

 Client fails to recover from this.  If one restarts client (sending same
 message again, see above), server never responds, so have to reboot HSM,
 after which everything works fine.

 Attaching to Berlin workshop milestone because this could be annoying
 during the workshop.

 First (easy) patch is to teach `cryptech_probe` to send more tailored
 nonsense down the RPC channel after identifying it, to clean up after the
 probe.

 Second (probably still easy) patch is to teach RPC client code to check
 for the expected opcode and discard malformed or unexpected packets.

 Don't yet understand what's going on in the HSM well enough to know what
 the fix is there.

--
Ticket URL: <https://trac.cryptech.is/ticket/63>
Open Crypto Project <https://wiki.cryptech.is/>