Update docs and replace some references to sfbuild/symbiflow with f4pga

Signed-off-by: Krzysztof Boronski <kboronski@antmicro.com>
This commit is contained in:
Krzysztof Boronski 2022-05-06 10:58:44 -05:00
parent 1e82b22bb5
commit c763733b8b
16 changed files with 153 additions and 155 deletions

View File

@ -1,38 +1,45 @@
# Developer's notes
##### Last update: 2022-05-06
:::{warning}
These notes are provided as-is and they shouldn't be treated as a full-blown accurate
documentation, but rather as a helpful resource for those who want to get involved with
development of _f4pga_. These are not updated regularly.
For more detailed, up-to-date information about the code, refer to the pydoc documentation.
:::
## Project's structure
The main script is in the `sfbuild.py` file.
`sf_cache.py` contains code needed for tracking modifications in the project.
`sf_ugly` contains some ugly workarounds.
* `__init__.py` contains the logic and entrypoint of the build system
* `argparser.py` contains boring code for CLI interface
* `cache.py` contains code needed for tracking modifications in the project.
* `common.py` contains code shared by the main utility and the modules
* `flow_config.py` contains code for reading and accessing flow definitions and configurations
* `module_inspector.py` contains utilities for inpecting I/O of modules
* `module_runner.py` contains code required to load modules at runtime
* `module.py` contains definitions required for writing and using f4pga modules
* `part_db.json` contains mappings from part names to platform names
* `setup.py` contains a package installation script
* `stage.py` contains classes relevant to stage represetation
* `modules` contains loadable modules
* `platforms` contains platform flow definitions
There a are two python modules which are shared by the code of `sfbuild.py` and
_sfbuild modules_: `sf_common` and `sf_module`.
:::{important}
Through the codebase _f4pga_ (tool) might be often referenced as _sfbuild_.
Similarly, _F4PGA_ (toolchain) might get called _Symbiflow_.
This is due to the project being written back when _F4PGA_ was called _Symbiflow_.
:::
_sfbuild modules_ are extensions to the build system that wrap tools to be used
within _sfbuild_ and currently they are standalone executable scripts. All
_sfbuild modules_ are single python scripts located under directories that
follow `sf_*_modules/` pattern. So currently those are:
* `sf_common_modules` - modules which can be shared by multiple platforms.
* `sf_xc7_modules` - modules specific to xc7 flows.
* `sf_quicklogic_modules` - modules specific to Quiclogic flows.
There's also a `docs` directory which you are probably aware of if you are reading
this. All the documentation regarding sfbuild goes here.
`platforms` direcotory contains JSON files with _platform flow definitions_.
Names of those files must follow `platform_name.json` pattern.
## Differnt subsystems and where to find them?
## Different subsystems and where to find them?
### Building and dependency resolution
All the code regarding dependency resolution is located in `sfbuild.py` file.
All the code regarding dependency resolution is located in `__init__.py` file.
Take a look at the `Flow` class.
Most of the work is done in `Flow._resolve_dependencies` method. Basically it
performs a _DFS_ with _stages_ (instances of _sfbuild modules_) as its nodes
performs a _DFS_ with _stages_ (instances of _f4pga modules_) as its nodes
which are linked using symbolic names of dependencies on inputs and outputs.
It queries the modules for information regarding i/o (most importantly the paths
on which they are going to produce outputs), checks whether
@ -54,27 +61,16 @@ to individual stages.
Keeping track of status of each file is done using `SymbiCache` class, which is
defined in `sf_cache.py` file. `SymbiCache` is used mostly inside `Flow`'s methods.
### Module's internals and API
`sf_module` contains everything that is necessary to write a module.
Prticularly the `Module` and `ModuleContext` classes
The `do_module` function currently servers as to create an instance of some
`Module`'s subtype and provide a _CLI_ interface for it.
The _CLI_ interface however, is not meant to be used by an end-user, especially
given that it reads JSON data from _stdin_. A wrapper for interfacing with modules
exists in `sfbuild.py` and it's called `_run_module`.
### Internal environmental variable system
_sfbuild_ exposes some data to the user as well as reads some using internal
_f4pga_ exposes some data to the user as well as reads some using internal
environmental variables. These can be referenced by users in
_platform flow definitions_ and _project flow configurations_ using the
`${variable_name}` syntax when defining values. They can also be read inside
_sfbuild modules_ by accesing the `ctx.values` namespace.
_f4pga modules_ by accesing the `ctx.values` namespace.
The core of tis system is the `ResolutionEnvironemt` class which can be found
inside the `sf_common` module.
inside the `common` module.
### Installation
@ -82,23 +78,6 @@ Check `CMakeLists.txt`.
## TODO:
Therea re a couple things that need some work:
### Urgent
* Full support for Quicklogic platforms.
* Testing XC7 projects with more sophisticated setups and PCF flows.
### Important
* Fix and refactor overloading mechanism in _platform flow definitions_ and
_platform flow configurations_. Values in the global `values` dict should
be overloaded by those in `values` dict under `module_options.stage_name`
inside _platform flow definition_. Values in `platform flow configuration`
should be imported from `platform flow definition` and then overloaded by
entries in `values`, `platform_name.values`,
`platform_name.stages.stage_name.values` dicts respectively.
* Define a clear specification for entries in _platform flow definitions_ and
_platform flow configurations_. Which environmental variables can be accessed
where, and when?
@ -110,64 +89,15 @@ Therea re a couple things that need some work:
* Make commenting style consistent
* Write more docs
### Not very important
* Document writing flow defintions
* Extend the metadata system for modules, perhaps make it easier to use.
* Add missing metadata for module targets.
### Maybe possible in the future
* Generate platform defintions using CMake.
* (_suggestion_) Generate platform defintions using CMake.
### Out of the current scope
* Change interfaces of some internal python scripts. This could lead to possibly
merging some modules for XC7 and Quicklogic into one common module.
## Quicklogic
So far I've been trying to bring support to _EOS-S3_ platform with mixed results.
Some parts of upstream Symbiflow aren't there yet. The Quicklogic scripts are
incomplete.
The _k4n8_ family remains a mystery to me. There's zero information about any
other familiar that _PP3_ and _PP2_. Neither could I find example projects for that.
Symbiflow's website mentions that only briefly. Yosys complains about `_DLATCH_N_`
not being supported when I tried synthesisng anything. Possibly related to the fact
that there's no equivalent of `pp3_latches_map.v` for `k4n8/umc22` in
[Yosys](https://github.com/YosysHQ/yosys/tree/master/techlibs/quicklogic).
**UPDATE**: Finally got the ioplace stage to work. Pulling the Quicklogic fork was
necessary in order to progress. The Quicklogic EOS-S3 development is now moved into
`eos-s3` branch of my fork.
Additionally The `chandalar.pcf` file in _symbiflow-examples_ seemed to be faulty.
The '()' parenthesis should be replaced by '[]' brackets.
I also tried to synthesize the `iir` project from `tool-perf`, but **VPR** seems
to be unable to fit it (at least on my installation of Symbiflow which at this point
is a bit old and heavily modified).
Here's a flow configuration I've used for `btn_counter` on `eos-s3`:
```json
{
"dependencies": {
"sources": ["btn_counter.v"],
"synth_log": "${build_dir}/synth.log",
"pack_log": "${build_dir}/pack.log"
},
"values": {
"top": "top",
"build_dir": "build/eos-s3"
},
"ql-eos-s3": {
"dependencies": {
"pcf": "chandalar.pcf",
"build_dir": "${build_dir}"
}
}
}
```

View File

@ -77,7 +77,8 @@ All *dependencies* are tracked by a modification tracking system which stores ha
When F4PGA constructs a *flow*, it will try to omit execution of modules which would receive the same data on their
input.
There is a strong _assumption_ there that a *module*'s output remains unchanged if the input configuration isn't
change, ie. *modules* are deterministic.
changed, ie. *modules* are deterministic. This is might be not true for some tools and in case you really want to re-run
a stage, there's a `--nocache` option that treats the `.symbicache` file as if it was empty.
### Resolution
@ -247,24 +248,21 @@ Project status:
[N] sources: ['counter.v']
[O] xdc: ['arty.xdc']
F4PGA: DONE
f4pga: DONE
```
The letters in the boxes describe the status of a dependency which's name is next to the box.
* **X** - dependency unresolved.
Dependency is not present or cannot be produced.
This isn't always a bad sign. Some dependencies are not required to, such as `pcf`.
* **U** - dependency unreachable.
The dependency has a module that could produce it, but the module's dependencies are unresolved.
This doesn't say whether the dependency was necessary or not.
* **O** - dependency present, unchanged.
This dependency is already built and is confirmed to stay unchanged during flow execution.
* **N** - dependency present, new/changed.
This dependency is already present on the persistent storage, but it was either missing earlier, or its content
changed from the last time.
changed since the last time it was used.
:::{warning}
It won't continue to be reported as "**N**" after a successful build of any target.
@ -292,11 +290,8 @@ Additional info about a dependency will be displayed next to its name after a co
* In case of unresolved dependencies (**X**), which are never produced by any module, a text sying "`MISSING`" will be
displayed.
* In case of unreachable dependencies, a name of such module that could produce them will be displayed followed by
`-> ???`.
In the example above file `counter.v` has been modified and is now marked as "**N**".
This couses a bunch of other dependencies to be reqbuilt ("**R**").
This causes a bunch of other dependencies to be reqbuilt ("**R**").
`build_dir` and `xdc` were already present, so they are marked as "**O**".
## Common targets and values
@ -328,7 +323,7 @@ Below are lists of the target and value names along with their meanings.
| Value name | type | Description |
|------------|------|-------------|
| `shareDir` | `string` | Path to symbiflow's installation "share" directory |
| `shareDir` | `string` | Path to f4pga's installation "share" directory |
| `python3` | `string` | Path to Python 3 executable |
| `noisyWarnings` | `string` | Path to noisy warnings log (should be deprecated) |
| `prjxray_db` | `string` | Path to Project X-Ray database |

View File

@ -2,6 +2,6 @@
MY_DIR=`dirname $0`
SFBUILD_DIR=${MY_DIR}/../../f4pga
SFBUILD_PY=${SFBUILD_DIR}/sfbuild.py
SFBUILD_PY=${SFBUILD_DIR}/__init__.py
PYTHONPATH=${SFBUILD_DIR} pydoc -b

View File

@ -0,0 +1,18 @@
# fasm
The _fasm_ module generates FPGA assebly using `genfasm` (VPR-only).
The module should guarantee the following outputs:
* `fasm`
For detailed information about these targets, please refer to
`docs/common targets and variables.md`
The setup of the synth module follows the following specifications:
## Values
The `fasm` module accepts the following values:
* `pnr_corner` (string, optional): PnR conrenr to use. Relevant only for Quicklogic's
eFPGAs.

View File

@ -1,6 +1,6 @@
# generic_script_wrapper
This module provides a way to integrate an external command into an sfbuild flow.
This module provides a way to integrate an external command into an f4pga flow.
Its inputs and outputs are fully defined by the author of flow definition.
## Parameters

View File

@ -3,20 +3,21 @@
## Interface
This document contains all the information needed to configure modules for
your _**sfbuild**_ project as well as some info about the API used to write
your _**f4pga**_ project as well as some info about the API used to write
modules.
### Configuration interface:
Modules are configured through an internal API by _**sfbuild**_.
Modules are configured through an internal API by _**sfbuf4pgaild**_.
The basic requirement for a module script is to expose a class with `Module`
interface.
_**sfbuild**_ reads configuration from two different places:
**platform's flow definition** file and **project's flow configuration** file.
_**f4pga**_ reads its configuration from two different sources:
**platform's flow definition**, which is a file that usually comes bundled with f4pga
and **project's flow configuration**, which is a set of configuration options provided by the user
through a JSON file or CLI interface.
The files, as described in "_Getting Started_" document, contain _JSON_ serialized
data. And they contain snippets of _module configurations_
Thosse sources contain snippets of _module configurations_.
A _module configuration_ is a structure with the following fields:
@ -24,15 +25,13 @@ A _module configuration_ is a structure with the following fields:
The values are paths to those dependencies.
They can be either singular strings or lists of strings.
* `produces` = a dictionary that contains keys which are names of the dependencies produced by the module.
* `produces` - a dictionary that contains keys which are names of the dependencies produced by the module.
The values are requested filenames for the files generated by the module.
They can be either singular strings or lists of strings.
* `values` - a dictionary that contains other values used to configure the module.
The keys are value's names and the values can have any type.
* `platform` - Platform's name. This is a string.
### Platform-level configuration
In case of **platform's flow definition**, a `values` dictionary can be defined
@ -40,25 +39,29 @@ globally and the values defined there will be passed to every module's config.
Those values can be overriden per-module through `module_options` dictionary.
Parameters used during module's contruction can also be defined in `module_options`
Parameters used during module's construction can also be defined in `module_options`
as `params` (those are not a part of _module configuration_, instead they are used
during the actual construction of a module instance, before it declares any of its
input/outputs etc.)
input/outputs etc.. This is typically used to acieve some parametrization over module's
I/O).
Defining dictionaries for `takes` and `produces` is disallowed within
Defining dictionaries for `takes` and `produces` is currently disallowed within
**platform's flow definition**.
For a detailed look on the concepts described here, please have a look at
`sfbuild/platforms/xc7a50t`
For examples of **platform's flow definition** described here, please have a look at
`f4pga/platforms/` directory. It contains **platform flow definitions** that come bundled
with f4pga.
### Project-level configuration
This section describes **project's flow configuration**.
Similarly to **platform's flow definition**, `values` dict can be provided.
The values provided there will overwrite the values from
**platform's flow definition** in case of a collision.
Unlike **platform's flow definition**, **project's flow configuration** may contain
`dependencies` dict. This dictionary would be used to map saymbolic dependency
`dependencies` dict. This dictionary would be used to map symbolic dependency
names to actual paths. Most dependencies can have their paths resolved implicitly
without the need to provide explicit paths, which is a mechanism that is described
in a later section of this document. However some dependencies must be provided
@ -71,24 +74,34 @@ dependencies, which won't be produced unless the user explicitelly provides a pa
for them.
**project's flow configuration** cannot specify `params` for modules and does not
use `module_options` dictionary.
use `module_options` dictionary. Neither it can instantiate any extra stages.
Any entry with a key other than `dependencies` or `values` is treated as a
platform name. Thise entries are necessaery to enable support for a given platform.
Any entry with a couple _exceptions*_ is treated as a platform name.
Enabling support for a given platform within a **project's flow configuration** file
requires having an entry for that platform.
Each of those entries may contain `dependencies`, `values` fields which will
overload the `dependecies` and `values` defined in a global scope of
**project's flow configuration**. Any other field under those platform entries
is treated as a _stage-specific-configuration_. The key is a name of a stage within
a flow for the specified platform and the values are dicts which may contain
`dependencies` and `values` fields that overload `dependencies` and `values`
repespectively, locally for the stage.
repespectively, locally for the stage. Additionally a `default_target` field can be
provided to specify a default target to built when the user does not specify it throgh
a CLI inteface.
The afromentioned _*exceptions_ are:
* `dependencies` - dependencies shared by all platforms.
* `values` - values shared by all platforms
* `default_platform` - default platform to chose in case it doesn't get specified
by the user
### Internal environmental variables
It's very usefule to be able to refer to some data within
It's very useful to be able to refer to some data within
**platform's flow definition** and **project's flow configuration** to
either avoid redundant definitions or to access results of certain operations.
_**sfbuild**_ allows doing that by using a special syntax for accessing internal
either avoid redundant definitions or to store and access results of certain operations.
_**f4pga**_ allows doing that by using a special syntax for accessing internal
environmental variables.
The syntax is `${variable_name}`. Any string value within
@ -148,7 +161,8 @@ categories:
(more on that later).
* **built-in references** - there are a couple of built-in variables which are very
handy:
* `shareDir` - path to symbiflow's _share_ directory.
* `shareDir` - path to f4pga's _share_ directory.
* `binDir` - path to f4pga's _bin_ directory.
* `prjxray_db` - Project X-Ray database path.
* `python3` - path to Python 3 interpreter.
* `noisyWarnings` - (this one should probably get removed)
@ -166,7 +180,7 @@ The class should implement the following methods:
is a dict with optional parameter for the module.
Each module script should expose the class by defining it's name/type alias as
`ModuleClass`. sfbuild tries to access a `ModuleClass` attribute within a package
`ModuleClass`. f4pga tries to access a `ModuleClass` attribute within a package
when instantiating a module.
### Module's execution modes
@ -224,7 +238,7 @@ set:
by the module.
* `values` - a list of names given to the variables used withing the module
* `prod_meta` - A dictionary which maps product names to descriptions of these
products.
products. Those entries are optional and can be skipped.
#### Qualifiers/decorators
@ -250,8 +264,13 @@ is implemented at the moment. It might be removed in the future.
## Common modules
```{toctree}
fasm
generic_script_wrapper
io_rename
mkdirs
pack
place
place_constraints
route
synth
```

View File

@ -19,3 +19,7 @@ Not specifying a mapping for a given entry will leave it with its original name.
## Values
All values specified for this modules will be accessible by tyhe wrapped module.
## Extra notes
This module might be removed in the future in favor of a native renaming support.

View File

@ -1,4 +1,4 @@
# io_rename
# mkdirs
This modules creates directiories specified by the author of flow definition
as its targets..

View File

@ -0,0 +1,7 @@
# pack
:::{warning}
this page is under construction
:::
Pack circuit with VPR.

View File

@ -0,0 +1,7 @@
# place
:::{warning}
this page is under construction
:::
Place cells with VPR.

View File

@ -0,0 +1,11 @@
# place_constraints
:::{warning}
this page is under construction
:::
Move cell placement to satisfy constraints imposed by an architecture. (VPR-only)
:::{note}
This will be deprecated once VPR constraint system supports this functionality natively.
:::

View File

@ -0,0 +1,7 @@
# route
:::{warning}
this page is under construction
:::
Route a design with VPR.

View File

@ -20,17 +20,17 @@ will be generated upon a successful YOSYS run.
The setup of the synth module follows the following specifications:
## Parameters:
## Parameters
The `params` section of a stage configuration may contain a `produces` list.
The list should specify additional targets that will be generated
(`?` qualifier is allowedd).
(`?` qualifier is allowed).
## Values:
## Values
The `synth` module requires the following values:
* `tcl_scripts` (string, required ): A path to a directory containing `synth.tcl`
* `tcl_scripts` (string, required): A path to a directory containing `synth.tcl`
and `conv.tcl` scripts that wiull be used by YOSYS.
* `read_verilog_args` (list[string | number], optional) - If specified, the verilog
sources will be read using the `read_verilog` procedure with options contained in

View File

@ -458,8 +458,8 @@ def sfbuild_fail():
sfbuild_done_str = Style.BRIGHT + Fore.RED + 'FAILED'
def sfbuild_done():
sfprint(1, f'sfbuild: {sfbuild_done_str}'
f'{Style.RESET_ALL + Fore.RESET}')
sfprint(1, f'f4pga: {sfbuild_done_str}'
f'{Style.RESET_ALL + Fore.RESET}')
exit(0)
def setup_resolution_env():

View File

@ -102,7 +102,7 @@ def _setup_build_parser(parser: ArgumentParser):
metavar='<name=path, ...>',
type=str,
help='Specify stage inputs explicitely. This might be required if some files got renamed or deleted and '
'symbiflow is unable to deduce the flow that lead to dependencies required by the requested stage'
'f4pga is unable to deduce the flow that lead to dependencies required by the requested stage'
)
@ -130,7 +130,7 @@ def setup_argparser():
"""
Set up argument parser for the program.
"""
parser = ArgumentParser(description='SymbiFlow Build System')
parser = ArgumentParser(description='F4PGA Build System')
parser.add_argument(
'-v',

View File

@ -56,8 +56,8 @@ class ModuleContext:
execution.
"""
share: str # Absolute path to Symbiflow's share directory
bin: str # Absolute path to Symbiflow's bin directory
share: str # Absolute path to F4PGA's share directory
bin: str # Absolute path to F4PGA's bin directory
takes: SimpleNamespace # Maps symbolic dependency names to relative paths.
produces: SimpleNamespace # Contains mappings for explicitely specified dependencies.
# Useful mostly for checking for on-demand optional outputs (such as logs) with