upsilon/doc/programmers_manual.md

Upsilon Programmers Manual. This document may be distributed under the terms
of the GNU GPL v3.0 (or any later version), or under the [CC BY-SA 4.0][CC].

[CC]: https://creativecommons.org/licenses/by-sa/4.0/legalcode

# Introduction

This document is aimed at maintainers of this software who are not
experienced programmers (in either software or hardware). Its goal is
to contain any pertinent information to the devlopment process of Upsilon.

This manual is (hopefully) modular enough that you can just skip to the
section you need without having to read the entire thing.

## Organization of the Project

Upsilon uses LiteX and ZephyrOS for it's FPGA code. LiteX generates HDL
and glues it together. It also forms the build system of the hardware
portion of Upsilon. ZephyrOS is the kernel portion, which deals with
communication between the computer that receives scan data and the
hardware that is executing the scan.

LiteX further uses F4PGA to compile the HDL code. F4PGA is primarily
made up of Yosys (synthesis) and nextpnr (place and route).

## Required Knowledge

Verilog is critical for writing hardware. You should hopefully not have
to write much of it.

The kernel is written in C. This C is different than C you have written
before because it is running "freestanding."

The kernel uses Zephyr as the real-time operating system running the
code. Zephyr has very good documentation and a very readable code base,
go read it.

Tests are written in C++ and verilog. You will not have to write C++
unless you modify the Verilog files.

The macro processing language GNU m4 is used occasionally. You will
need to know how to use m4 if you modify the main `base.v.m4` file
(e.g. adding more software-accessable ports).

Python is used for the bytecode assembler, the bytecode itself, and
the SoC generator. The SoC generator uses a library called LiteX,
which in turn uses migen.
You do not need to know a lot about migen, but LiteX's documentation
is poor so you will need to know some migen in order to read the
code and understand how some modules work.

# Compile Process

Although each component uses a different build system, you can run everything
with
1. `make` (compile everything in this folder)
2. `make clean` (clean up all compiled files)

## Setting up the Toolchain

The toolchain is primarily designed around modern Linux. It may not work
properly on Windows or MacOS. If you have access to a computational
cluster (if you are at FSU physics, ask the Physics department) then
you should set up the toolchain on their servers. You will be able to
compile things on any computer with an internet connection.

### F4PGA

1. Clone [F4PGA](https://github.com/chipsalliance/f4pga) (if you want,
   checkout commit `b6c5fff`, but you should try checking out master
   first)
2. Run `scripts/prepare_environment.sh`. Note that you will need to change
   the environment variable `$F4PGA_INSTALL_DIR` if you do not have access
   to the default directory (which is root access).
3. Run `scripts/activate.sh`. If you run into problems, open the file and
   copy the `source` and `conda` commands manually into your terminal.
4. Install meson and ninja through pip.

All commands should be done in the conda environment.

### Zephyr OS

These instructions are based on [these][zephyr_getting_started], but the Zephyr
environment should be installed into the F4PGA conda environment,

[zephyr_getting_started]: https://docs.zephyrproject.org/latest/develop/getting_started/index.html

1. Run `pip3 install west`
2. Run
   ```
   west init $ZEPHYR_DIRECTORY/zephyrproject
   cd $ZEPHYR_DIRECTORY/zephyrproject
   west update
   west zephyr-export
   pip install -r ~/zephyrproject/zephyr/scripts/requirements.txt
   cd $ZEPHYR_DIRECTORY/zephyrproject
   wget https://github.com/zephyrproject-rtos/sdk-ng/releases/download/v0.16.0/zephyr-sdk-0.16.0_linux-x86_64.tar.xz
   wget -O - https://github.com/zephyrproject-rtos/sdk-ng/releases/download/v0.16.0/sha256.sum | shasum --check --ignore-missing
   tar xvf zephyr-sdk-0.16.0_linux-x86_64.tar.xz
   cd zephyr-sdk-0.16.0
   ./setup.sh
   ```

### LiteX

1. Download `litex_setup.py` from the [LiteX repository][litex_repo], Upsilon
   uses 2022.08 to some directory (don't put it in your home directory because
   there will be a bunch of downloaded repositories.
2. Run `litex_setup.py --init --install --user --tag 2022.08`
3. Download a GCC RISC-V cross compiler. If you have root access to the build
   machine, then you can probably install this with your package manager. Users
   of Ubuntu 14 can download the [sifive][sifive_gcc] GCC. Otherwise you will have
   to compile a cross compiler (`x86_64` host to RV32I target) manually.
4. Put the GCC RISC-V cross compiler in your `$PATH` variable.

[litex_repo]: https://github.com/enjoy-digital/litex
[sifive_gcc]: https://github.com/sifive/freedom-tools/releases

## FPGA Build System

Make sure F4PGA and a RISC-V GCC compiler are in your path. Then just go into
the `firmware` folder and run `make`. This should generate everything you need
and compile the software. The synthesis suite is single threaded. This will
take about 15-20 minutes on a good computer.

The FPGA firmware (aka gateware) build system is designed in a recursive
manner. That means that each directory has a Makefile that processes all the
files in the directory. There is a `common.makefile` in the `rtl/` directory
that is used when a rule (such as preprocessing a Verilog source file)
is used in multiple Makefiles.

For the Arty A7, the bitstream is `firmware/build/digilent_arty/gateware/digilent_arty.bit`.

## Software Build System

The software build system uses files that are generated by the FPGA compile
process. The number one reason why software won't work when loaded onto the
FPGA is because it is compiled for a different FPGA bitstream. If you have
an issue where software that you know should work does not, run `make clean`
in the FPGA build system and rebuild it from scratch.

This requires at least CMake 3.20.0 (you can install this using `conda`).
Afterwards just run `make` and everything should work. Everything is
managed by the `CMakeLists.txt` and the `prj.conf`, see the Zephyr OS
documentation.

The kernel is `/software/build/zephyr/zephyr.bin`

If you make a change to `CMakeLists.txt` or to `prj.conf`, run `make clean`
before `make`.

Make can run in parallel using `-j${NUMBER_OF_PROCESSORS}`. Add this to the
`buidl/zephyr/zephyr.bin` in `/software/Makefile` to makeyour builds faster.
Remove this argument when you are attemping to fix compile errors and warnings
(it will make the build output easier to read) but put it back when you fix
them.

# Loading the Software and Firmware

## Network Setup

You will need the FPGA and the controlling computer on the same wired
network. **DO NOT CONNECT THE FPGA TO A WIDE NETWORK. USE A PRIVATE LAN
THAT ONLY CONTAINS THE CONTROLLING COMPUTER AND THE FPGA. DO NOT ATTEMPT
TO CONNECT THE FPGA TO THE INTERNET.** The controlling computer can
still connect to the internet, but through another LAN port. The best
thing to do is to buy a USB to Ethernet adapter.

You will need some way to do DHCP. The best way is to use a router, but
a standard wireless router will not fly with any IT department because
of the security risk. You need to find a non-wireless router (like a
managed switch). You can even retrofit an old computer into a router
(just needs another ethernet port).

The default TFTP client connects to 192.168.1.50.

## Connecting to the FPGA Over USB

Connect to the FPGA over USB and run `litex_term /dev/ttyUSB1` (or whatever
connection it should be) and you should see the LiteX BIOS come up. 

## Loading the Firmware

Connect the FPGA to a computer using a Micro-USB to USB cable. Run
`openFPGALoader -c digilent digilent_arty.bit` to upload the firmware
(gateware) to the controller.

You can load the software using serial boot but this is very slow. The
better thing to do is to use TFTP boot, which goes over Ethernet.
**WHEN YOU RUN TFTP, DO NOT EXPOSE YOUR INTERFACE TO THE INTERNET
CONNECTED NETWORK INTERFACE. THIS IS A BIG SECURITY RISK. ONLY RUN
TFTP FOR THE AMOUNT OF TIME REQUIRED TO BOOT THE CONTROL SOFTWARE.**
You can read about how to setup a TFTP server on the [OpenWRT wiki][owrt_wiki].
On Linux, run

	dnsmasq --port=0 --enable-tftp --tftp-root=/path/to/firmware/directory --user=root --group=root --interface=$INTERFACE

Do not use `--tftp-no-blocksize`. The controller will only read the first
512 bytes of the kernel.

In the root of the TFTP server, have `boot.bin` be the kernel binary
(`zephyr.bin`).

[owrt_wiki]: https://openwrt.org/docs/guide-user/troubleshooting/tftpserver

# FPGA

Upsilon runs on a Field Programmable Gate Array (FPGA). FPGAs are sets
of logic gates and other peripherals that can be changed by a computer.
FPGAs can implement CPUs, digital filters, and control code at a much
higher speed than a computer. The downside is that FPGAs are much more
difficult to program for.

A large part of Upsilon is written in Verilog. Verilog is a Hardware
Description Language (HDL), which is similar to a programming language
(such as C++ or Python).

The difference is, is that Verilog compiles to a *piece of hardware* that
deals with individual bits executing operations in sync with a clock. This
differs from a *piece of software*, which is a set of instructions that a
computer follows. Verilog is usually much less abstract than regular code.

Regular code is tested on the system in which it is run. Hardware,
on the other hand, is very difficult to test on the device that it
is actually running on. Hardware is usually *simulated*. This project
primarily simulates Verilog code using the program Verilator, where the
code that runs the simulation is written in C++.

Instead of strings, integers, and classes, the basic components of all
Verilog code is the wire and the register, which store bits (1 and 0).
Wires connect components together, and registers store data, in a similar
way to variables in software. Unlike usual programming languages, where
code executes one step at a time, most FPGA code runs at the tick of
the system clock in parallel.

To compile Verilog to a format suitable for execution on an FPGA, you
*synthesize* the Verilog into a low-level format that uses the specific
resources of the FPGA you are using, and then you run a *place and route*
program to allocate resources on the FPGA to fit your design. Running
synthesis on its own can help you understand how much resources a module
uses. Place-and-route gives you *timing reports*, which tell you about
major design problems that outstrip the capabilities of the FPGA (or the
programs you are using). You should look up what "timing" on an FPGA is
and learn as much as you can about it, because it is an issue that does
not happen in standard software and can be very difficult to fix when
you run into it.

Once a bitstream is synthesized, it is loaded onto a FPGA through a cable
(for this project, openFPGALoader).

## Recommendations to Learners

[Gisselquist Technology][GT] is the best free online resource for FPGA
programming out there. These articles will help you understand how to
write *good* FPGA code, not just valid code.

[GT]: https://zipcpu.com/

Here are some exercises for you to ease yourself into FPGA programming.

* Write an FPGA program that implements addition without using the `+`
  operator. This program should add each number bit by bit, handling
  carried digits properly. This is called a *full adder*.
* Write an FPGA program that multiplies two signed integers together,
  without using the `*` operator.  The width of these integers should
  not be hard-coded: it should be easy to change. What you write in
  this is something that is actually a part of this project: see
  `boothmul.v`. You do not (and should not!) write it just like Upsilon
  has written it.
* Write an FPGA program that communicates over SPI. For simplicity,
  you only need to write it for a single SPI mode: look up on the internet
  for details. There is an SPI slave device in this repository that you
  can use to simulate an end for the SPI master you write, but you should
  write the SPI slave yourself. For bonus points, connect your SPI master
  to a real SPI device and confirm that your communication works.

For each of these exercises, follow the complete "Design Testing Process"
below. At the very least, write simulations and test your programs on
real hardware.

## Control and Status Registers in Hardware

LiteX uses "Control and Status Registers" (CSRs) to communicate between
the CPU and any Verilog modules. (RISC-V CPUs have something with the
same name, but Upsilon does not use that.)

## Design Testing Process

### Simulation

When you write or modify a verilog module, the first thing you should do
is write/run a simulation of that module. A simulation of that module
should at the minimum compare the execution of the module with known
results (called "Ground truth testing"). A simulation should also consider
edge cases that you might overlook when writing Verilog.

For example, a module that multiplies two signed integers together should
have a simulation that sends the module many pairs of integers, taking
care to ensure that all possible permutations of sign are tested (i.e.
positive times positive, negative times positive, etc.) and also that
special-cases are handled (i.e. largest 32-bit integer multiplied by
largest negative 32-bit integer, multiplication by 0 and 1, etc.).

Writing simulation code is a very boring task, but you *must* do it.
Otherwise there is no way for you to check that

1. Your code does what you want it to do
2. Any changes you make to your code don't break it

If you find a bug that isn't covered by your simulation, make sure you
add that case to the simulation.

The file `firmware/rtl/testbench.hpp` contains a class that you should
use to organize individual tests. Make a derived class of `TB` and
use the `posedge()` function to encode what default actions your test
should take at every positive edge of the clock. Remember, in C++ each
action is blocking: there is no equivalent to the non-blocking `<=`.

If you have to do a lot of non-blocking code for your test, you
should write a Verilog wrapper for your test that implements
the non-blocking code. **Verilator only supports a subset of
non-synthesizable Verilog.  Unless you really need to, use synthesizable
Verilog only.** See `firmware/rtl/waveform/waveform_sim.v` and
`firmware/rtl/waveform/dma_sim.v` for an example of Verilog files only
used for tests.

### Test Synthesis

**Yosys only accepts a subset of Verilog. You might write a bunch of
code that Verilator will happily simulate but that will fail to go
through Yosys.**

Once you have simulated your design, you should use yosys to synthesize it.
This will allow you to understand how much and what resources the module
is taking up. To do this, you can put the follwing in a script file:

    read_verilog module_1.v
    read_verilog module_2.v
    ...
    read_verilog top_module.v
    synth_xilinx -flatten -nosrl -noclkbuf -nodsp -iopad -nowidelut
    write_verilog yosys_synth_output.v

and run `yosys -s scriptfile`. The options to `synth_xilinx` reflect
the current limitations that F4PGA has. The file `xc7.f4pga.tcl` that
F4PGA downloads is the complete synthesis script, read it to understand
the internals of what F4PGA does to compile your verilog.

### Test Compilation

I haven't been able to do this for most of this project. The basic idea
is to use `firmware/rtl/soc.py` to load only the module to test, and
to use LiteScope to write and read values from the module. For more
information, you can look at
[the boothmul test](https://software.mcgoron.com/peter/boothmul/src/branch/master/arty_test).

# Software Programming

The "software" is the code written in C that runs on the FPGA. This
handles access to hardware components, running scripts sent by the
controlling computer, and sending information between the hardware and
the controlling computer.

## Crash Course in Multithreaded Programming

Each script (up to 32 by default, change by redefining a macro) runs in
a separate thread. This allows for multiple scripts to execute without
having to explicitly hand control from one component to another, but
since there is no defined execution path (one thread may execute before
or after another thread), the program must handle scripts attempting to
access the same component.

Upsilon handles multiple threads using

1. Mutexes
2. Thread Local Storage

Mutexes ("mutual exclusion") are objects that only allow for one thread
to access them at a time. When one thread locks a mutex, other threads
attempting to lock the mutex sleep until the thread unlocks the mutex.
After the thread that locked the mutex unlocks it, some other thread gets
the mutex.

Mutex management is important because if multiple threads attempt to
read or write to a converter at the same time, the scripts could deadlock,
requiring a hard reset of the system. (You could add manual deadlock
aborting by adding new commands that call `k_thread_abort`, as long as
all threads are not deadlocked. This is a hack but may be necessary.)

Each thread can lock the mutex as many times as it wants, but it must
unlock the mutex the same number of times. Thread local storage (the
`__thread` modifier) is used to count the number of times that each mutex
is locked by a thread. Since (as the name implies) TLS is thread-local,
there is no need to control access to it by mutexes: each thread gets
its own local version of the thread local variables.

The software has to count the number of recursive locks because when
the thread finally releases control of the mutex, another thread must
be able to access the hardware in a well defined state: it should not
attempt to write to hardware while the hardware is running (certain
specific exceptions apply). When the unlock routines (see for example
`waveform_release()`) reach the final unlock
(e.g. `waveform_locked[i] == 1`), the software waits for the hardware
to finish what its doing before unlocking.

## Crash Course in Network Programming

The kernel communicates with the controlling computer using a TCP/IP
connection. You should connect the controller and the computer to a
router and assign the kernel a static IP.

Each script that runs on the kernel is a separate connection. Each
connection runs on a separate thread, because each thread runs a Creole
interpreter.

TODO

## Logging

Logging is done via UART. Connect the micro-USB slot to the controlling
computer to get debug output.

All you need to know is

* Use `LOG_WRN` for errors that you can recover from (i.e. closing a
  connection
* Use `LOG_ERR` for errors that are fatal and halt the firmware,
  requiring a reset
* Use `LOG_INF` for misc information (i.e. initialization completed,
  accepted connection, closing connection)
* Use `LOG_DBG` for debugging output

If you need debugging output, add a line of the form

	set_source_file_properties(src_file PROPERTIES COMPILE_FLAGS -DFILE_LOG_LEVEL=4

This will enable debugging output for this file only. Do not enable
debugging output for the entire system! This will make the debugging
output unusuable.

When you are done, set `4` to `3` in that line.

TODO: Ethernet debugging output.

## Control and Status Registers in Software

CSR read and write functions are generated by `/firmware/generate_csr_locations.py`.
You should not need to directly call `write` and `read` on raw addresses.
If you add a new CSR, add it to the generator script.

### Implementation Information

CSRs can be used in software by using `litex_write8`,
`litex_read16`, etc. In the Zephyr source, look at
`soc/riscv/litex-vexriscv/soc.h` for the complete implementation.
Also look at `include/zephyr/arch/common/sys_io.h` to see how these
functions are implemented.

Do not directly write to CSR ports without using `litex_writeN` and
`litex_readN`, and do not directly use `sys_io.h` functions. If you are
not careful you will not access the registers correctly and you will
crash the software.

# Controlling Computer

## Creole

Creole is the bytecode that the kernel runs. It is written using a
python library. It looks very similar to assembly, but is custom built
to make it easier to write direct assembly code.

Creole programs are the scripts run by the kernel to communicate with
hardware and send messages over Ethernet to the controlling computer.
Each creole program should do one thing: i.e. monitor an ADC, run
the raster scan, output waveforms, etc.

Creole programs should reserve the hardware modules (DAC, ADC, CLOOP,
waveforms) that they use explicitly. This makes your program faster
and less error prone.

Since the Creole assembler is a python library, you can use things
like Python format strings to automate production of Creole code. You
can also add virtual instructions (by directly modifying the library)
easily.

Creole has a concept of data blocks, assigned using the `DB` command.
These blocks are used for waveforms and for printing sets of data out
to the datastream.

Creole uses a [self-synchronizing code][ssc] to detect encoding and
transmission errors. This makes programs bigger, but you should not
write big Creole programs.

[ssc]: https://en.wikipedia.org/wiki/Self-synchronizing_code

# Hacks and Pitfalls

The open source software stack that Upsilon uses is novel and unstable.

## F4PGA

This is really a Yosys (and really, an abc bug). F4PGA defaults to using
the ABC flow, which can break, especially for block RAM. To fix, edit out
`-abc` in the tcl script (find it before you install it...)

## Yosys

Yosys fails to calculate computed parameter values correctly. For instance,

    parameter CTRLVAL = 5;
    localparam VALUE = CTRLVAL + 1;

Yosys will *silently* fail to compile this, setting `VALUE` to be equal
to 0. The solution is to use macros.

## Macros

Verilog's preprocessor is awful. F4PGA (through yosys) barely supports it.

You should only use Verilog macros as a replacement for `localparam`.
When you need to do so, you must preprocess the file with
Verilator. For example, if you have a file called `mod.v` in the folder
`firmware/rtl/mod/`, then in the file `firmware/rtl/mod/Makefile` add

    codegen: [...] mod_preprocessed.v

(putting it after all other generated files). The file
`firmware/rtl/common.makefile` should automatically generate the
preprocessed file for you.

## If The Controlling Computer Cannot Connect to the Internet

When you connect your computer to the controller over Ethernet, your computer
may attempt to route all traffic over the controller network (since it is
wired) instead of another network (like a wireless network). This means that
your computer can't connect to the internet (or your connection is really slow).
If this happens to you on a Linux machine, you can change the routing table.

Run `route -n` (or `ip route` if this does not work) to print the routing table.
Find the entry named `default via [...] dev eth-interface`. This is the default route
for the ethernet device. Remove it using `ip route del default via [...] dev eth-interface`.

If the route keeps on reappearing, delete it and quickly enter
`ip route del default via [...] dev eth0 metric 65534`. This will make the
route the last priority.

## Getting The Correct IP for the Controlling Computer

Some routers can automatically assign IPs based on MAC address. If your computer
can do that, great. Otherwise you will need to configure your computer with a
static ip.

1. Remove your computer from the DHCP list that the router has.
2. Run `ip link set eth-interface up`.
3. Then run `ip addr` and run `ip addr remove del [ip] dev eth-interface` on
   each ip on the ethernet interface that is connected to the controller.
3. Run `ip addr add 192.168.1.100/24 dev eth-interface` (or whatever ip + subnet
   mask you need)
4. Run `ip route add 192.168.1.0/24 dev eth0 proto kernel scope link` (again,
   change depending on different situations)

This will use the static ip `192.168.1.100`, which is the default TFTP boot
IP.