Update docs and replace some references to sfbuild/symbiflow with f4pga

Signed-off-by: Krzysztof Boronski <kboronski@antmicro.com>

docs/f4pga/DevNotes.md

@@ -1,38 +1,45 @@
 # Developer's notes
+##### Last update: 2022-05-06
+
+:::{warning}
+These notes are provided as-is and they shouldn't be treated as a full-blown accurate
+documentation, but rather as a helpful resource for those who want to get involved with
+development of _f4pga_. These are not updated regularly.
+
+For more detailed, up-to-date information about the code, refer to the pydoc documentation.
+:::
 
 ## Project's structure
 
-The main script is in the `sfbuild.py` file.
+* `__init__.py` contains the logic and entrypoint of the build system
-`sf_cache.py` contains code needed for tracking modifications in the project.
+* `argparser.py` contains boring code for CLI interface
-`sf_ugly` contains some ugly workarounds.
+* `cache.py` contains code needed for tracking modifications in the project.
+* `common.py` contains code shared by the main utility and the modules
+* `flow_config.py` contains code for reading and accessing flow definitions and configurations
+* `module_inspector.py` contains utilities for inpecting I/O of modules
+* `module_runner.py` contains code required to load modules at runtime
+* `module.py` contains definitions required for writing and using f4pga modules
+* `part_db.json` contains mappings from part names to platform names
+* `setup.py` contains a package installation script
+* `stage.py` contains classes relevant  to stage represetation
+* `modules` contains loadable modules
+* `platforms` contains platform flow definitions
 
-There a are two python modules which are shared by the code of `sfbuild.py` and
+:::{important}
-_sfbuild modules_: `sf_common` and `sf_module`.
+Through the codebase _f4pga_ (tool) might be often referenced as _sfbuild_.
+Similarly, _F4PGA_ (toolchain) might get called _Symbiflow_.
+This is due to the project being written back when _F4PGA_ was called _Symbiflow_.
+:::
 
-_sfbuild modules_ are extensions to the build system that wrap tools to be used
+## Different subsystems and where to find them?
-within _sfbuild_ and currently they are standalone executable scripts. All
-_sfbuild modules_ are single python scripts located under directories that
-follow `sf_*_modules/` pattern. So currently those are:
-
- * `sf_common_modules` - modules which can be shared by multiple platforms.
- * `sf_xc7_modules` - modules specific to xc7 flows.
- * `sf_quicklogic_modules` - modules specific to Quiclogic flows.
-
-There's also a `docs` directory which you are probably aware of if you are reading
-this. All the documentation regarding sfbuild goes here.
-
-`platforms` direcotory contains JSON files with _platform flow definitions_.
-Names of those files must follow `platform_name.json` pattern.
-
-## Differnt subsystems and where to find them?
 
 ### Building and dependency resolution
 
-All the code regarding dependency resolution is located in `sfbuild.py` file.
+All the code regarding dependency resolution is located in `__init__.py` file.
 Take a look at the `Flow` class.
 
 Most of the work is done in `Flow._resolve_dependencies` method. Basically it
-performs a _DFS_ with _stages_ (instances of _sfbuild modules_) as its nodes
+performs a _DFS_ with _stages_ (instances of _f4pga modules_) as its nodes
 which are linked using symbolic names of dependencies on inputs and outputs.
 It queries the modules for information regarding i/o (most importantly the paths
 on which they are going to produce outputs), checks whether
@@ -54,27 +61,16 @@ to individual stages.
 Keeping track of status of each file is done using `SymbiCache` class, which is
 defined in `sf_cache.py` file. `SymbiCache` is used mostly inside `Flow`'s methods.
 
-### Module's internals and API
-
-`sf_module` contains everything that is necessary to write a module.
-Prticularly the `Module` and `ModuleContext` classes 
-The `do_module` function currently servers as to create an instance of some
-`Module`'s subtype and provide a _CLI_ interface for it.
-
-The _CLI_ interface however, is not meant to be used by an end-user, especially
-given that it reads JSON data from _stdin_. A wrapper for interfacing with modules
-exists in `sfbuild.py` and it's called `_run_module`.
-
 ### Internal environmental variable system
 
-_sfbuild_ exposes some data to the user as well as reads some using internal
+_f4pga_ exposes some data to the user as well as reads some using internal
 environmental variables. These can be referenced by users in
 _platform flow definitions_ and _project flow configurations_ using the
 `${variable_name}` syntax when defining values. They can also be read inside
-_sfbuild modules_ by accesing the `ctx.values` namespace.
+_f4pga modules_ by accesing the `ctx.values` namespace.
 
 The core of tis system is the `ResolutionEnvironemt` class which can be found
-inside the `sf_common` module.
+inside the `common` module.
 
 ### Installation
 
@@ -82,23 +78,6 @@ Check `CMakeLists.txt`.
 
 ## TODO:
 
-Therea re a couple things that need  some work:
-
-### Urgent
-
-* Full support for Quicklogic platforms.
-* Testing XC7 projects with more sophisticated setups and PCF flows.
-
-### Important
-
-* Fix and refactor overloading mechanism in _platform flow definitions_ and
-  _platform flow configurations_. Values in the global `values` dict should
-  be overloaded by those in `values` dict under `module_options.stage_name`
-  inside _platform flow definition_. Values in `platform flow configuration`
-  should be imported from `platform flow definition` and then overloaded by
-  entries in `values`, `platform_name.values`,
-  `platform_name.stages.stage_name.values` dicts respectively.
-
 * Define a clear specification for entries in _platform flow definitions_ and
   _platform flow configurations_. Which environmental variables can be accessed
   where, and when?
@@ -110,64 +89,15 @@ Therea re a couple things that need  some work:
 
 * Make commenting style consistent
 
-* Write more docs
+* Document writing flow defintions
-
-### Not very important
 
 * Extend the metadata system for modules, perhaps make it easier to use.
 
 * Add missing metadata for module targets.
 
-### Maybe possible in the future
+* (_suggestion_) Generate platform defintions using CMake.
-
-* Generate platform defintions using CMake.
 
 ### Out of the current scope
 
 * Change interfaces of some internal python scripts. This could lead to possibly
   merging some modules for XC7 and Quicklogic into one common module.
-
-## Quicklogic
-
-So far I've been trying to bring support to _EOS-S3_ platform with mixed results.
-Some parts of upstream Symbiflow aren't there yet. The Quicklogic scripts are
-incomplete.
-
-The _k4n8_ family remains a mystery to me. There's zero information about any
-other familiar that _PP3_ and _PP2_. Neither could I find example projects for that.
-Symbiflow's website mentions that only briefly. Yosys complains about `_DLATCH_N_`
-not being supported when I tried synthesisng anything. Possibly related to the fact
-that there's no equivalent of `pp3_latches_map.v` for `k4n8/umc22` in
-[Yosys](https://github.com/YosysHQ/yosys/tree/master/techlibs/quicklogic).
-
-**UPDATE**: Finally got the ioplace stage to work. Pulling the Quicklogic fork was
-necessary in order to progress. The Quicklogic EOS-S3 development is now moved into
-`eos-s3` branch of my fork. 
-Additionally The `chandalar.pcf` file in _symbiflow-examples_ seemed to be faulty.
-The '()' parenthesis should be replaced by '[]' brackets.
-I also tried to synthesize the `iir` project from `tool-perf`, but **VPR** seems
-to be unable to fit it (at least on my installation of Symbiflow which at this point
-is a bit old and heavily modified).
-
-Here's a flow configuration I've used for `btn_counter` on `eos-s3`:
-
-```json
-{
-    "dependencies": {
-        "sources": ["btn_counter.v"],
-        "synth_log": "${build_dir}/synth.log",
-        "pack_log": "${build_dir}/pack.log"
-    },
-    "values": {
-        "top": "top",
-        "build_dir": "build/eos-s3"
-    },
-
-    "ql-eos-s3": {
-        "dependencies": {
-            "pcf": "chandalar.pcf",
-            "build_dir": "${build_dir}"
-        }
-    }
-}
-```

docs/f4pga/Usage.md

@@ -77,7 +77,8 @@ All *dependencies* are tracked by a modification tracking system which stores ha
 When F4PGA constructs a *flow*, it will try to omit execution of modules which would receive the same data on their
 input.
 There is a strong _assumption_ there that a *module*'s output remains unchanged if the input configuration isn't
-change, ie. *modules* are deterministic.
+changed, ie. *modules* are deterministic. This is might be not true for some tools and in case you really want to re-run
+a stage, there's a `--nocache` option that treats the `.symbicache` file as if it was empty.
 
 ### Resolution
 
@@ -247,24 +248,21 @@ Project status:
     [N] sources:  ['counter.v']
     [O] xdc:  ['arty.xdc']
 
-F4PGA: DONE
+f4pga: DONE
 ```
 
 The letters in the boxes describe the status of a dependency which's name is next to the box.
 
  * **X** - dependency unresolved.
+   Dependency is not present or cannot be produced.
    This isn't always a bad sign. Some dependencies are not required to, such as `pcf`.
 
- * **U** - dependency unreachable.
-   The dependency has a module that could produce it, but the module's dependencies are unresolved.
-   This doesn't say whether the dependency was necessary or not.
-
  * **O** - dependency present, unchanged.
    This dependency is already built and is confirmed to stay unchanged during flow execution.
 
  * **N** - dependency present, new/changed.
    This dependency is already present on the persistent storage, but it was either missing earlier, or its content
-   changed from the last time.
+   changed since the last time it was used.
 
    :::{warning}
    It won't continue to be reported as "**N**" after a successful build of any target.
@@ -292,11 +290,8 @@ Additional info about a dependency will be displayed next to its name after a co
 * In case of unresolved dependencies (**X**), which are never produced by any module, a text sying "`MISSING`" will be
   displayed.
 
-* In case of unreachable dependencies, a name of such module that could produce them will be displayed followed by
-  `-> ???`.
-
 In the example above file `counter.v` has been modified and is now marked as "**N**".
-This couses a bunch of other dependencies to be reqbuilt ("**R**").
+This causes a bunch of other dependencies to be reqbuilt ("**R**").
 `build_dir` and `xdc` were already present, so they are marked as "**O**".
 
 ## Common targets and values
@@ -328,7 +323,7 @@ Below are lists of the target and value names along with their meanings.
 
 | Value name | type | Description |
 |------------|------|-------------|
-| `shareDir` | `string` | Path to symbiflow's installation "share" directory |
+| `shareDir` | `string` | Path to f4pga's installation "share" directory |
 | `python3` | `string` | Path to Python 3 executable |
 | `noisyWarnings` | `string` | Path to noisy warnings log (should be deprecated) |
 | `prjxray_db` | `string` | Path to Project X-Ray database |

docs/f4pga/browse_pydoc.sh

@@ -2,6 +2,6 @@
 
 MY_DIR=`dirname $0`
 SFBUILD_DIR=${MY_DIR}/../../f4pga
-SFBUILD_PY=${SFBUILD_DIR}/sfbuild.py
+SFBUILD_PY=${SFBUILD_DIR}/__init__.py
 
 PYTHONPATH=${SFBUILD_DIR} pydoc -b

docs/f4pga/modules/fasm.md

@@ -0,0 +1,18 @@
+# fasm
+
+The _fasm_ module generates FPGA assebly using `genfasm` (VPR-only).
+
+The module should guarantee the following outputs:
+ * `fasm`
+
+For detailed information about these targets, please refer to
+`docs/common targets and variables.md`
+
+The setup of the synth module follows the following specifications:
+
+## Values
+
+The `fasm` module accepts the following values:
+
+* `pnr_corner` (string, optional): PnR conrenr to use. Relevant only for Quicklogic's
+  eFPGAs.

docs/f4pga/modules/generic_script_wrapper.md

@@ -1,6 +1,6 @@
 # generic_script_wrapper
 
-This module provides a way to integrate an external command into an sfbuild flow.
+This module provides a way to integrate an external command into an f4pga flow.
 Its inputs and outputs are fully defined by the author of flow definition.
 
 ## Parameters

docs/f4pga/modules/index.md

@@ -3,20 +3,21 @@
 ## Interface
 
 This document contains all the information needed to configure modules for
-your _**sfbuild**_ project as well as some info about the API used to write
+your _**f4pga**_ project as well as some info about the API used to write
 modules.
 
 ### Configuration interface:
 
-Modules are configured through an internal API by _**sfbuild**_.
+Modules are configured through an internal API by _**sfbuf4pgaild**_.
 The basic requirement for a module script is to expose a class with `Module`
 interface.
 
-_**sfbuild**_ reads configuration from two different places:
+_**f4pga**_ reads its configuration from two different sources:
-**platform's flow definition** file and **project's flow configuration** file.
+**platform's flow definition**, which is a file that usually comes bundled with f4pga
+and **project's flow configuration**, which is a set of configuration options provided by the user
+through a JSON file or CLI interface.
 
-The files, as described in "_Getting Started_" document, contain _JSON_ serialized
+Thosse sources contain snippets of _module configurations_.
-data. And they contain snippets of _module configurations_
 
 A _module configuration_ is a structure with the following fields:
 
@@ -24,15 +25,13 @@ A _module configuration_ is a structure with the following fields:
   The values are paths to those dependencies.
   They can be either singular strings or lists of strings.
 
-* `produces` = a dictionary that contains keys which are names of the dependencies produced by the module.
+* `produces` - a dictionary that contains keys which are names of the dependencies produced by the module.
   The values are requested filenames for the files generated by the module.
   They can be either singular strings or lists of strings.
 
 * `values` - a dictionary that contains other values used to configure the module.
   The keys are value's names and the values can have any type.
 
-* `platform` - Platform's name. This is a string.
-
 ### Platform-level configuration
 
 In case of **platform's flow definition**, a `values` dictionary can be defined
@@ -40,25 +39,29 @@ globally and the values defined there will be passed to every module's config.
 
 Those values can be overriden per-module through `module_options` dictionary.
 
-Parameters used during module's contruction can also be defined in `module_options`
+Parameters used during module's construction can also be defined in `module_options`
 as `params` (those are not a part of _module configuration_, instead they are used
 during the actual construction of a module instance, before it declares any of its
-input/outputs etc.)
+input/outputs etc.. This is typically used to acieve some parametrization over module's
+I/O).
 
-Defining dictionaries for `takes` and `produces` is disallowed within
+Defining dictionaries for `takes` and `produces` is currently disallowed within
 **platform's flow definition**.
 
-For a detailed look on the concepts described here, please have a look at
+For examples of **platform's flow definition** described here, please have a look at
-`sfbuild/platforms/xc7a50t`
+`f4pga/platforms/` directory. It contains **platform flow definitions** that come bundled
+with f4pga. 
 
 ### Project-level configuration
 
+This section describes **project's flow configuration**.
+
 Similarly to **platform's flow definition**, `values` dict can be provided.
 The values provided there will overwrite the values from
 **platform's flow definition** in case of a collision.
 
 Unlike **platform's flow definition**, **project's flow configuration** may contain
-`dependencies` dict. This dictionary would be used to map saymbolic dependency
+`dependencies` dict. This dictionary would be used to map symbolic dependency
 names to actual paths. Most dependencies can have their paths resolved implicitly
 without the need to provide explicit paths, which is a mechanism that is described
 in a later section of this document. However some dependencies must be provided
@@ -71,24 +74,34 @@ dependencies, which won't be produced unless the user explicitelly provides a pa
 for them.
 
 **project's flow configuration** cannot specify `params` for modules and does not
-use `module_options` dictionary.
+use `module_options` dictionary. Neither it can instantiate any extra stages.
 
-Any entry with a key other than `dependencies` or `values` is treated as a
+Any entry with a couple _exceptions*_ is treated as a platform name.
-platform name. Thise entries are necessaery to enable support for a given platform.
+Enabling support for a given platform within a **project's flow configuration** file
+requires having an entry for that platform.
 Each of those entries may contain `dependencies`, `values` fields which will
 overload the `dependecies` and `values` defined in a global scope of
 **project's flow configuration**. Any other field under those platform entries
 is treated as a _stage-specific-configuration_. The key is a name of a stage within
 a flow for the specified platform and the values are dicts which may contain
 `dependencies` and `values` fields that overload `dependencies` and `values`
-repespectively, locally for the stage.
+repespectively, locally for the stage. Additionally a `default_target` field can be
+provided to specify a default target to built when the user does not specify it throgh
+a CLI inteface.
+
+The afromentioned _*exceptions_ are:
+
+* `dependencies` - dependencies shared by all platforms.
+* `values` - values shared by all platforms
+* `default_platform` - default platform to chose in case it doesn't get specified
+  by the user
 
 ### Internal environmental variables
 
-It's very usefule to be able to refer to some data within
+It's very useful to be able to refer to some data within
 **platform's flow definition** and **project's flow configuration** to
-either avoid redundant definitions or to access results of certain operations.
+either avoid redundant definitions or to store and access results of certain operations.
-_**sfbuild**_ allows doing that by using a special syntax for accessing internal
+_**f4pga**_ allows doing that by using a special syntax for accessing internal
 environmental variables.
 
 The syntax is `${variable_name}`. Any string value within
@@ -148,7 +161,8 @@ categories:
   (more on that later).
 * **built-in references** - there are a couple of built-in variables which are very
   handy:
-  * `shareDir` - path to symbiflow's _share_ directory.
+  * `shareDir` - path to f4pga's _share_ directory.
+  * `binDir` - path to f4pga's _bin_ directory.
   * `prjxray_db` - Project X-Ray database path.
   * `python3` - path to Python 3 interpreter.
   * `noisyWarnings` - (this one should probably get removed)
@@ -166,7 +180,7 @@ The class should implement the following methods:
   is a dict with optional parameter for the module.
 
 Each module script should expose the class by defining it's name/type alias as
-`ModuleClass`. sfbuild tries to access a `ModuleClass` attribute within a package
+`ModuleClass`. f4pga tries to access a `ModuleClass` attribute within a package
 when instantiating a module.
 
 ### Module's execution modes
@@ -224,7 +238,7 @@ set:
   by the module.
 * `values` - a list of names given to the variables used withing the module
 * `prod_meta` - A dictionary which maps product names to descriptions of these
-  products.
+  products. Those entries are optional and can be skipped.
 
 #### Qualifiers/decorators
 
@@ -250,8 +264,13 @@ is implemented at the moment. It might be removed in the future.
 ## Common modules
 
 ```{toctree}
+fasm
 generic_script_wrapper
 io_rename
 mkdirs
+pack
+place
+place_constraints
+route
 synth
 ```

docs/f4pga/modules/io_rename.md

@@ -19,3 +19,7 @@ Not specifying a mapping for a given entry will leave it with its original name.
 ## Values
 
 All values specified for this modules will be accessible by tyhe wrapped module.
+
+## Extra notes
+
+This module might be removed in the future in favor of a native renaming support.

docs/f4pga/modules/mkdirs.md

@@ -1,4 +1,4 @@
-# io_rename
+# mkdirs
 
 This modules creates directiories specified by the author of flow definition
 as its targets..

docs/f4pga/modules/pack.md

@@ -0,0 +1,7 @@
+# pack
+
+:::{warning}
+this page is under construction
+:::
+
+Pack circuit with VPR.

docs/f4pga/modules/place.md

@@ -0,0 +1,7 @@
+# place
+
+:::{warning}
+this page is under construction
+:::
+
+Place cells with VPR.

docs/f4pga/modules/place_constraints.md

@@ -0,0 +1,11 @@
+# place_constraints
+
+:::{warning}
+this page is under construction
+:::
+
+Move cell placement to satisfy constraints imposed by an architecture. (VPR-only)
+
+:::{note}
+This will be deprecated once VPR constraint system supports this functionality natively.
+:::

docs/f4pga/modules/route.md

@@ -0,0 +1,7 @@
+# route
+
+:::{warning}
+this page is under construction
+:::
+
+Route a design with VPR.

docs/f4pga/modules/synth.md

@@ -20,17 +20,17 @@ will be generated upon a successful YOSYS run.
 
 The setup of the synth module follows the following specifications:
 
-## Parameters:
+## Parameters
 
 The `params` section of a stage configuration may contain a `produces` list.
 The list should specify additional targets that will be generated
-(`?` qualifier is allowedd).
+(`?` qualifier is allowed).
 
-## Values:
+## Values
 
 The `synth` module requires the following values:
 
-* `tcl_scripts` (string, required ): A path to a directory containing `synth.tcl`
+* `tcl_scripts` (string, required): A path to a directory containing `synth.tcl`
   and `conv.tcl` scripts that wiull be used by YOSYS.
 * `read_verilog_args` (list[string | number], optional) - If specified, the verilog
   sources will be read using the `read_verilog` procedure with options contained in

f4pga/__init__.py

@@ -458,7 +458,7 @@ def sfbuild_fail():
     sfbuild_done_str = Style.BRIGHT + Fore.RED + 'FAILED'
 
 def sfbuild_done():
-    sfprint(1, f'sfbuild: {sfbuild_done_str}'
+    sfprint(1, f'f4pga: {sfbuild_done_str}'
                f'{Style.RESET_ALL + Fore.RESET}')
     exit(0)
 

f4pga/argparser.py

@@ -102,7 +102,7 @@ def _setup_build_parser(parser: ArgumentParser):
         metavar='<name=path, ...>',
         type=str,
         help='Specify stage inputs explicitely. This might be required if some files got renamed or deleted and '
-             'symbiflow is unable to deduce the flow that lead to dependencies required by the requested stage'
+             'f4pga is unable to deduce the flow that lead to dependencies required by the requested stage'
     )
 
 
@@ -130,7 +130,7 @@ def setup_argparser():
     """
     Set up argument parser for the program.
     """
-    parser = ArgumentParser(description='SymbiFlow Build System')
+    parser = ArgumentParser(description='F4PGA Build System')
 
     parser.add_argument(
         '-v',

f4pga/module.py

@@ -56,8 +56,8 @@ class ModuleContext:
     execution.
     """
 
-    share: str                 #  Absolute path to Symbiflow's share directory
+    share: str                 #  Absolute path to F4PGA's share directory
-    bin: str                   #  Absolute path to Symbiflow's bin directory
+    bin: str                   #  Absolute path to F4PGA's bin directory
     takes: SimpleNamespace     #  Maps symbolic dependency names to relative paths.
     produces: SimpleNamespace  #  Contains mappings for explicitely specified dependencies.
                                #  Useful mostly for checking for on-demand optional outputs (such as logs) with