Hardware
/
VexRiscv
mirror of https://github.com/SpinalHDL/VexRiscv.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Improve testing infrastructure with more options and better readme 

https://github.com/litex-hub/linux-on-litex-vexriscv/issues/112
This commit is contained in:
Dolu1990 2020-03-01 13:02:08 +01:00
parent 0c255e2404
commit 559260020b
2 changed files with 80 additions and 41 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

45
README.md
View File

@ -186,24 +186,47 @@ NOTES:
[![Build Status](https://travis-ci.org/SpinalHDL/VexRiscv.svg?branch=master)](https://travis-ci.org/SpinalHDL/VexRiscv) [![Build Status](https://travis-ci.org/SpinalHDL/VexRiscv.svg?branch=master)](https://travis-ci.org/SpinalHDL/VexRiscv)
To run tests (need the verilator simulator), go in the src/test/cpp/regression folder and run : To run tests (need the java, scala, verilator), just do :
```sh ```sh
# To test the GenFull CPU export VEXRISCV_REGRESSION_SEED=42
# (Don't worry about the CSR test not passing, basicaly the GenFull isn't the truly full version of the CPU, some CSR features are disable in it) export VEXRISCV_REGRESSION_TEST_ID=
make clean run sbt "testOnly vexriscv.TestIndividualFeatures"
# To test the GenSmallest CPU
make clean run IBUS=SIMPLE DBUS=SIMPLE CSR=no MMU=no DEBUG_PLUGIN=no MUL=no DIV=no
``` ```
The self-test includes: This will generate random VexRiscv configuration and test them with:
- ISA tests from https://github.com/riscv/riscv-tests/tree/master/isa - ISA tests from https://github.com/riscv/riscv-tests/tree/master/isa and https://github.com/riscv/riscv-compliance
- Dhrystone benchmark - Dhrystone benchmark
- 24 FreeRTOS tests - Coremark benchmark
- Zephyr os
- Buildroot/Linux os
- Some handwritten tests to check the CSR, debug module and MMU plugins - Some handwritten tests to check the CSR, debug module and MMU plugins
You can enable FreeRTOS tests by adding `FREERTOS=yes` to the command line, but it will take time to run. Also, it uses THREAD_COUNT host CPU threads to run multiple regression in parallel. You can rerun some specific test by setting VEXRISCV_REGRESSION_TEST_ID by their id. For instance, if you want to rerun :
- test_id_5_test_IBus_CachedS1024W1BPL32Relaxvexriscv.plugin.DYNAMIC_DBus_CachedS8192W2BPL16_MulDiv_MulDivFpga_Shift_FullLate_Branch_Late_Hazard_BypassAll_RegFile_SyncDR_Src__Csr_AllNoException_Decoder__Debug_None_DBus_NoMmu
- test_id_9_test_IBus_Simple1S2InjStagevexriscv.plugin.STATIC_DBus_SimpleLate_MulDiv_MulDivFpgaSimple_Shift_FullEarly_Branch_Late_Hazard_Interlock_RegFile_AsyncER_Src_AddSubExecute_Csr_None_Decoder__Debug_None_DBus_NoMmu
then :
```
export VEXRISCV_REGRESSION_TEST_ID=5,9
```
Also there is a few environnement variable that you can use to modulate the random generation :
| Parameters | range | description |
| ------------------------------------------- | ------------------ | ----------- |
| VEXRISCV_REGRESSION_SEED | Int | Seed used to generate the random configurations |
| VEXRISCV_REGRESSION_TEST_ID | \[Int\[,\Int\]\*\] | Random configuration that should be keeped and tested |
| VEXRISCV_REGRESSION_CONFIG_COUNT | Int | Number of random configurations |
| VEXRISCV_REGRESSION_CONFIG_RVC_RATE | 0.0-1.0 | Chance to generate a RVC config |
| VEXRISCV_REGRESSION_CONFIG_LINUX_RATE | 0.0-1.0 | Chance to generate a linux ready config |
| VEXRISCV_REGRESSION_CONFIG_MACHINE_OS_RATE | 0.0-1.0 | Chance to generate a machine mode OS ready config |
| VEXRISCV_REGRESSION_LINUX_REGRESSION | yes/no | Enable the linux test |
| VEXRISCV_REGRESSION_COREMARK | yes/no | Enable the Coremark test |
| VEXRISCV_REGRESSION_ZEPHYR_COUNT | Int | Number of zephyr tests to run on capable configs |
| VEXRISCV_REGRESSION_CONFIG_DEMW_RATE | 0.0-1.0 | Chance to generate a config with writeback stage |
| VEXRISCV_REGRESSION_CONFIG_DEM_RATE | 0.0-1.0 | Chance to generate a config with memory stage |
## Interactive debug of the simulated CPU via GDB OpenOCD and Verilator ## Interactive debug of the simulated CPU via GDB OpenOCD and Verilator
It's as described to run tests, but you just have to add `DEBUG_PLUGIN_EXTERNAL=yes` in the make arguments. It's as described to run tests, but you just have to add `DEBUG_PLUGIN_EXTERNAL=yes` in the make arguments.

76
src/test/scala/vexriscv/TestIndividualFeatures.scala
View File

@ -3,7 +3,7 @@ package vexriscv
import java.io.File import java.io.File
import org.apache.commons.io.FileUtils import org.apache.commons.io.FileUtils
import org.scalatest.FunSuite import org.scalatest.{BeforeAndAfterAll, FunSuite, ParallelTestExecution}
import spinal.core._ import spinal.core._
import vexriscv.demo._ import vexriscv.demo._
import vexriscv.ip.{DataCacheConfig, InstructionCacheConfig} import vexriscv.ip.{DataCacheConfig, InstructionCacheConfig}
@ -313,7 +313,7 @@ class SrcDimension extends VexRiscvDimension("Src") {
} }
class IBusDimension extends VexRiscvDimension("IBus") { class IBusDimension(rvcRate : Double) extends VexRiscvDimension("IBus") {
override def randomPositionImpl(universes: Seq[ConfigUniverse], r: Random) = { override def randomPositionImpl(universes: Seq[ConfigUniverse], r: Random) = {
@ -322,7 +322,7 @@ class IBusDimension extends VexRiscvDimension("IBus") {
if(r.nextDouble() < 0.5){ if(r.nextDouble() < 0.5){
val latency = r.nextInt(5) + 1 val latency = r.nextInt(5) + 1
val compressed = r.nextBoolean() val compressed = r.nextDouble() < rvcRate
val injectorStage = r.nextBoolean() || latency == 1 val injectorStage = r.nextBoolean() || latency == 1
val prediction = random(r, List(NONE, STATIC, DYNAMIC, DYNAMIC_TARGET)) val prediction = random(r, List(NONE, STATIC, DYNAMIC, DYNAMIC_TARGET))
val catchAll = universes.contains(VexRiscvUniverse.CATCH_ALL) val catchAll = universes.contains(VexRiscvUniverse.CATCH_ALL)
@ -345,7 +345,7 @@ class IBusDimension extends VexRiscvDimension("IBus") {
} }
} else { } else {
val catchAll = universes.contains(VexRiscvUniverse.CATCH_ALL) val catchAll = universes.contains(VexRiscvUniverse.CATCH_ALL)
val compressed = r.nextBoolean() val compressed = r.nextDouble() < rvcRate
val tighlyCoupled = r.nextBoolean() && !catchAll val tighlyCoupled = r.nextBoolean() && !catchAll
// val tighlyCoupled = false // val tighlyCoupled = false
val prediction = random(r, List(NONE, STATIC, DYNAMIC, DYNAMIC_TARGET)) val prediction = random(r, List(NONE, STATIC, DYNAMIC, DYNAMIC_TARGET))
@ -497,14 +497,14 @@ class MmuDimension extends VexRiscvDimension("DBus") {
trait CatchAllPosition trait CatchAllPosition
class CsrDimension(freertos : String, zephyr : String) extends VexRiscvDimension("Csr") { class CsrDimension(freertos : String, zephyr : String, linux : String) extends VexRiscvDimension("Csr") {
override def randomPositionImpl(universes: Seq[ConfigUniverse], r: Random) = { override def randomPositionImpl(universes: Seq[ConfigUniverse], r: Random) = {
val catchAll = universes.contains(VexRiscvUniverse.CATCH_ALL) val catchAll = universes.contains(VexRiscvUniverse.CATCH_ALL)
val supervisor = universes.contains(VexRiscvUniverse.SUPERVISOR) val supervisor = universes.contains(VexRiscvUniverse.SUPERVISOR)
if(supervisor){ if(supervisor){
new VexRiscvPosition("Supervisor") with CatchAllPosition{ new VexRiscvPosition("Supervisor") with CatchAllPosition{
override def applyOn(config: VexRiscvConfig): Unit = config.plugins += new CsrPlugin(CsrPluginConfig.linuxFull(0x80000020l)) override def applyOn(config: VexRiscvConfig): Unit = config.plugins += new CsrPlugin(CsrPluginConfig.linuxFull(0x80000020l))
override def testParam = s"FREERTOS=$freertos ZEPHYR=$zephyr LINUX_REGRESSION=${sys.env.getOrElse("VEXRISCV_REGRESSION_LINUX_REGRESSION", "yes")} SUPERVISOR=yes" override def testParam = s"FREERTOS=$freertos ZEPHYR=$zephyr LINUX_REGRESSION=$linux SUPERVISOR=yes"
} }
} else if(catchAll){ } else if(catchAll){
new VexRiscvPosition("MachineOs") with CatchAllPosition{ new VexRiscvPosition("MachineOs") with CatchAllPosition{
@ -554,10 +554,35 @@ class DecoderDimension extends VexRiscvDimension("Decoder") {
} }
//class TesterPlay extends FunSuite with ParallelTestExecution {
// def createTest(name : String): Unit ={
// test(name){
// for(i <- 0 to 4) {
// println(s"$name $i")
// Thread.sleep(2000)
// }
// }
// }
// List("a", "b","c").foreach(createTest)
//}
class TestIndividualFeatures extends FunSuite { class TestIndividualFeatures extends FunSuite {
val testCount = sys.env.getOrElse("VEXRISCV_REGRESSION_CONFIG_COUNT", "100").toInt
val seed = sys.env.getOrElse("VEXRISCV_REGRESSION_SEED", Random.nextLong().toString).toLong
val testId : Set[Int] = sys.env.get("VEXRISCV_REGRESSION_TEST_ID") match {
case Some(x) if x != "" => x.split(',').map(_.toInt).toSet
case _ => (0 until testCount).toSet
}
val rvcRate = sys.env.getOrElse("VEXRISCV_REGRESSION_CONFIG_RVC_RATE", "0.5").toDouble
val linuxRate = sys.env.getOrElse("VEXRISCV_REGRESSION_CONFIG_LINUX_RATE", "0.3").toDouble
val machineOsRate = sys.env.getOrElse("VEXRISCV_REGRESSION_CONFIG_MACHINE_OS_RATE", "0.5").toDouble
val linuxRegression = sys.env.getOrElse("VEXRISCV_REGRESSION_LINUX_REGRESSION", "yes")
val coremarkRegression = sys.env.getOrElse("VEXRISCV_REGRESSION_COREMARK", "yes")
val zephyrCount = sys.env.getOrElse("VEXRISCV_REGRESSION_ZEPHYR_COUNT", "4")
val demwRate = sys.env.getOrElse("VEXRISCV_REGRESSION_CONFIG_DEMW_RATE", "0.6").toDouble
val demRate = sys.env.getOrElse("VEXRISCV_REGRESSION_CONFIG_DEM_RATE", "0.5").toDouble
def doCmd(cmd: String): String = { def doCmd(cmd: String): String = {
val stdOut = new StringBuilder() val stdOut = new StringBuilder()
class Logger extends ProcessLogger { class Logger extends ProcessLogger {
@ -578,7 +603,7 @@ class TestIndividualFeatures extends FunSuite {
val dimensions = List( val dimensions = List(
new IBusDimension, new IBusDimension(rvcRate),
new DBusDimension, new DBusDimension,
new MulDivDimension, new MulDivDimension,
new ShiftDimension, new ShiftDimension,
@ -586,7 +611,7 @@ class TestIndividualFeatures extends FunSuite {
new HazardDimension, new HazardDimension,
new RegFileDimension, new RegFileDimension,
new SrcDimension, new SrcDimension,
new CsrDimension(/*sys.env.getOrElse("VEXRISCV_REGRESSION_FREERTOS_COUNT", "1")*/ "0", sys.env.getOrElse("VEXRISCV_REGRESSION_ZEPHYR_COUNT", "4")), //Freertos old port software is broken new CsrDimension(/*sys.env.getOrElse("VEXRISCV_REGRESSION_FREERTOS_COUNT", "1")*/ "0", zephyrCount, linuxRegression), //Freertos old port software is broken
new DecoderDimension, new DecoderDimension,
new DebugDimension, new DebugDimension,
new MmuDimension new MmuDimension
@ -612,15 +637,15 @@ class TestIndividualFeatures extends FunSuite {
} }
val name = (if(noMemory) "noMemoryStage_" else "") + (if(noWriteback) "noWritebackStage_" else "") + positionsToApply.map(d => d.dimension.name + "_" + d.name).mkString("_") val name = (if(noMemory) "noMemoryStage_" else "") + (if(noWriteback) "noWritebackStage_" else "") + positionsToApply.map(d => d.dimension.name + "_" + d.name).mkString("_")
test(prefix + name + "_gen") { test(prefix + "gen_" + name) {
gen gen
} }
test(prefix + name + "_test") { test(prefix + "test_" + name) {
println("START TEST " + prefix + name) println("START TEST " + prefix + name)
val debug = true val debug = true
val stdCmd = (s"make clean run WITH_USER_IO=no REDO=10 TRACE=${if(debug) "yes" else "no"} TRACE_START=9999924910246l STOP_ON_ERROR=no FLOW_INFO=no STOP_ON_ERROR=no DHRYSTONE=yes COREMARK=${sys.env.getOrElse("VEXRISCV_REGRESSION_COREMARK", "yes")} THREAD_COUNT=${sys.env.getOrElse("VEXRISCV_REGRESSION_THREAD_COUNT", 1)} ") + s" SEED=${testSeed} " val stdCmd = (s"make clean run WITH_USER_IO=no REDO=10 TRACE=${if(debug) "yes" else "no"} TRACE_START=1000000000000l STOP_ON_ERROR=no FLOW_INFO=no STOP_ON_ERROR=no DHRYSTONE=yes COREMARK=${coremarkRegression} THREAD_COUNT=1 ") + s" SEED=${testSeed} "
val testCmd = stdCmd + (positionsToApply).map(_.testParam).mkString(" ") val testCmd = stdCmd + (positionsToApply).map(_.testParam).mkString(" ")
println(testCmd) println(testCmd)
val str = doCmd(testCmd) val str = doCmd(testCmd)
@ -628,42 +653,33 @@ class TestIndividualFeatures extends FunSuite {
} }
} }
val testId : Option[mutable.HashSet[Int]] = None
val seed = sys.env.getOrElse("VEXRISCV_REGRESSION_SEED", Random.nextLong().toString).toLong
//
// val testId = Some(mutable.HashSet(3,4,9,11,13,16,18,19,20,21))
// val testId = Some(mutable.HashSet(11))
// val testId = Some(mutable.HashSet(4, 11))
// val seed = 6592877339343561798l
val rand = new Random(seed) val rand = new Random(seed)
test("Info"){ test("Info"){
println(s"MAIN_SEED=$seed") println(s"MAIN_SEED=$seed")
} }
println(s"Seed=$seed") println(s"Seed=$seed")
for(i <- 0 until sys.env.getOrElse("VEXRISCV_REGRESSION_CONFIG_COUNT", "100").toInt){ for(i <- 0 until testCount){
var positions : List[VexRiscvPosition] = null var positions : List[VexRiscvPosition] = null
var universe = mutable.HashSet[VexRiscvUniverse]() var universe = mutable.HashSet[VexRiscvUniverse]()
if(rand.nextDouble() < 0.5) universe += VexRiscvUniverse.EXECUTE_RF if(rand.nextDouble() < 0.5) universe += VexRiscvUniverse.EXECUTE_RF
if(sys.env.getOrElse("VEXRISCV_REGRESSION_CONFIG_LINUX_RATE", "0.3").toDouble > rand.nextDouble()) { if(linuxRate > rand.nextDouble()) {
universe += VexRiscvUniverse.CATCH_ALL universe += VexRiscvUniverse.CATCH_ALL
universe += VexRiscvUniverse.MMU universe += VexRiscvUniverse.MMU
universe += VexRiscvUniverse.FORCE_MULDIV universe += VexRiscvUniverse.FORCE_MULDIV
universe += VexRiscvUniverse.SUPERVISOR universe += VexRiscvUniverse.SUPERVISOR
if(sys.env.getOrElse("VEXRISCV_REGRESSION_CONFIG_DEMW_RATE", "0.6").toDouble < rand.nextDouble()){ if(demwRate < rand.nextDouble()){
universe += VexRiscvUniverse.NO_WRITEBACK universe += VexRiscvUniverse.NO_WRITEBACK
} }
} else { } else {
if(sys.env.getOrElse("VEXRISCV_REGRESSION_CONFIG_MACHINE_OS_RATE", "0.5").toDouble > rand.nextDouble()) { if(machineOsRate > rand.nextDouble()) {
universe += VexRiscvUniverse.CATCH_ALL universe += VexRiscvUniverse.CATCH_ALL
if(sys.env.getOrElse("VEXRISCV_REGRESSION_CONFIG_DEMW_RATE", "0.6").toDouble < rand.nextDouble()){ if(demwRate < rand.nextDouble()){
universe += VexRiscvUniverse.NO_WRITEBACK universe += VexRiscvUniverse.NO_WRITEBACK
} }
} }
if(sys.env.getOrElse("VEXRISCV_REGRESSION_CONFIG_DEMW_RATE", "0.6").toDouble > rand.nextDouble()){ if(demwRate > rand.nextDouble()){
}else if(sys.env.getOrElse("VEXRISCV_REGRESSION_CONFIG_DEM_RATE", "0.5").toDouble > rand.nextDouble()){ }else if(demRate > rand.nextDouble()){
universe += VexRiscvUniverse.NO_WRITEBACK universe += VexRiscvUniverse.NO_WRITEBACK
} else { } else {
universe += VexRiscvUniverse.NO_WRITEBACK universe += VexRiscvUniverse.NO_WRITEBACK
@ -676,8 +692,8 @@ class TestIndividualFeatures extends FunSuite {
}while(!positions.forall(_.isCompatibleWith(positions))) }while(!positions.forall(_.isCompatibleWith(positions)))
val testSeed = rand.nextInt() val testSeed = rand.nextInt()
if(testId.isEmpty || testId.get.contains(i)) if(testId.contains(i))
doTest(positions," random_" + i + "_", testSeed, universe) doTest(positions," test_id_" + i + "_", testSeed, universe)
Hack.dCounter += 1 Hack.dCounter += 1
} }
} }