5. Manager Configuration Files¶
This page contains a centralized reference for all of the configuration options
in config_runtime.yaml
, config_build.yaml
, config_build_farm.yaml
,
config_build_recipes.yaml
, and config_hwdb.yaml
. It also contains
references for all build and run farm recipes (in deploy/build-farm-recipes and deploy/run-farm-recipes).
5.1. config_runtime.yaml
¶
Here is a sample of this configuration file:
# RUNTIME configuration for the FireSim Simulation Manager
# See https://docs.fires.im/en/stable/Advanced-Usage/Manager/Manager-Configuration-Files.html for documentation of all of these params.
run_farm:
# managerinit replace start
base_recipe: run-farm-recipes/aws_ec2.yaml
# Uncomment and add args to override defaults.
# Arg structure should be identical to the args given
# in the base_recipe.
#recipe_arg_overrides:
# <ARG>: <OVERRIDE>
# managerinit replace end
metasimulation:
metasimulation_enabled: false
# vcs or verilator. use vcs-debug or verilator-debug for waveform generation
metasimulation_host_simulator: verilator
# plusargs passed to the simulator for all metasimulations
metasimulation_only_plusargs: "+fesvr-step-size=128 +max-cycles=100000000"
# plusargs passed to the simulator ONLY FOR vcs metasimulations
metasimulation_only_vcs_plusargs: "+vcs+initreg+0 +vcs+initmem+0"
target_config:
topology: no_net_config
no_net_num_nodes: 1
link_latency: 6405
switching_latency: 10
net_bandwidth: 200
profile_interval: -1
# This references a section from config_hwdb.yaml for fpga-accelerated simulation
# or from config_build_recipes.yaml for metasimulation
# In homogeneous configurations, use this to set the hardware config deployed
# for all simulators
default_hw_config: firesim_rocket_quadcore_no_nic_l2_llc4mb_ddr3
# Advanced: Specify any extra plusargs you would like to provide when
# booting the simulator (in both FPGA-sim and metasim modes). This is
# a string, with the contents formatted as if you were passing the plusargs
# at command line, e.g. "+a=1 +b=2"
plusarg_passthrough: ""
tracing:
enable: no
# Trace output formats. Only enabled if "enable" is set to "yes" above
# 0 = human readable; 1 = binary (compressed raw data); 2 = flamegraph (stack
# unwinding -> Flame Graph)
output_format: 0
# Trigger selector.
# 0 = no trigger; 1 = cycle count trigger; 2 = program counter trigger; 3 =
# instruction trigger
selector: 1
start: 0
end: -1
autocounter:
read_rate: 0
workload:
workload_name: linux-uniform.json
terminate_on_completion: no
suffix_tag: null
host_debug:
# When enabled (=yes), Zeros-out FPGA-attached DRAM before simulations
# begin (takes 2-5 minutes).
# In general, this is not required to produce deterministic simulations on
# target machines running linux. Enable if you observe simulation non-determinism.
zero_out_dram: no
# If disable_synth_asserts: no, simulation will print assertion message and
# terminate simulation if synthesized assertion fires.
# If disable_synth_asserts: yes, simulation ignores assertion firing and
# continues simulation.
disable_synth_asserts: no
# DOCREF START: Synthesized Prints
synth_print:
# Start and end cycles for outputting synthesized prints.
# They are given in terms of the base clock and will be converted
# for each clock domain.
start: 0
end: -1
# When enabled (=yes), prefix print output with the target cycle at which the print was triggered
cycle_prefix: yes
# DOCREF END: Synthesized Prints
Below, we outline each mapping in detail.
5.1.1. run_farm
¶
The run_farm
mapping specifies the characteristics of your FireSim run farm so that the
manager can automatically launch them, run workloads on them, and terminate
them.
5.1.1.1. base_recipe
¶
The base_recipe
key/value pair specifies the default set of arguments to use for a particular run farm type.
To change the run farm type, a new base_recipe
file must be provided from deploy/run-farm-recipes
.
You are able to override the arguments given by a base_recipe
by adding keys/values to the recipe_arg_overrides
mapping.
5.1.1.2. recipe_arg_overrides
¶
This optional mapping of keys/values allows you to override the default arguments provided by the base_recipe
.
This mapping must match the same mapping structure as the args
mapping within the base_recipe
file given.
Overridden arguments override recursively such that all key/values present in the override args replace the default arguments given
by the base_recipe
. In the case of sequences, a overridden sequence completely replaces the corresponding sequence in the default args.
Additionally, it is not possible to change the default run farm type through these overrides.
This must be done by changing the default base_recipe
.
See Run Farm Recipes (run-farm-recipes/*) for more details on the potential run farm recipes that can be used.
5.1.2. metasimulation
¶
The metasimulation
options below allow you to run metasimulations
instead of FPGA simulations when doing launchrunfarm
, infrasetup
, and runworkload
.
See Debugging & Testing with Metasimulation for more details.
5.1.2.1. metasimulation_enabled
¶
This is a boolean to enable running metasimulations in-place of FPGA-accelerated simulations.
The number of metasimulations that are run on a specific Run Farm host is determined by the num_metasims
argument
in each run farm recipe (see Run Farm Recipes (run-farm-recipes/*)).
5.1.2.2. metasimulation_host_simulator
¶
This key/value pair chooses which RTL simulator should be used for metasimulation.
Options include verilator
and vcs
if waveforms are unneeded and *-debug
versions
if a waveform is needed.
5.1.2.3. metasimulation_only_plusargs
¶
This key/value pair is a string that passes plusargs (arguments with a +
in front) to the metasimulations.
5.1.2.4. metasimulation_only_vcs_plusargs
¶
This key/value pair is a string that passes plusargs (arguments with a +
in front) to metasimulations using vcs
or vcs-debug
.
5.1.3. target_config
¶
The target_config
options below allow you to specify the high-level
configuration of the target you are simulating. You can change these parameters
after launching a Run Farm (assuming you have the correct number of instances),
but in many cases you will need to re-run the infrasetup
command to make
sure the correct simulation infrastructure is available on your instances.
5.1.3.1. topology
¶
This field dictates the network topology of the simulated system. Some examples:
no_net_config
: This runs N (see no_net_num_nodes
below) independent
simulations, without a network simulation. You can currently only use this
option if you build one of the NoNIC hardware configs of FireSim.
example_8config
: This requires a single f1.16xlarge
, which will
simulate 1 ToR switch attached to 8 simulated servers.
example_16config
: This requires two f1.16xlarge
instances and one
m4.16xlarge
instance, which will
simulate 2 ToR switches, each attached to 8 simulated servers, with the two
ToR switches connected by a root switch.
example_64config
: This requires eight f1.16xlarge
instances and one
m4.16xlarge
instance, which will simulate 8 ToR switches, each attached to
8 simulated servers (for a total of 64 nodes), with the eight ToR switches
connected by a root switch.
Additional configurations are available in deploy/runtools/user_topology.py
and more can be added there. See the Manager Network Topology Definitions (user_topology.py) section
for more info.
5.1.3.2. no_net_num_nodes
¶
This determines the number of simulated nodes when you are using
topology: no_net_config
.
5.1.3.3. link_latency
¶
In a networked simulation, this allows you to specify the link latency of the simulated network in CYCLES. For example, 6405 cycles is roughly 2 microseconds at 3.2 GHz. A current limitation is that this value (in cycles) must be a multiple of 7. Furthermore, you must not exceed the buffer size specified in the NIC’s simulation widget.
5.1.3.4. switching_latency
¶
In a networked simulation, this specifies the minimum port-to-port switching latency of the switch models, in CYCLES.
5.1.3.5. net_bandwidth
¶
In a networked simulation, this specifies the maximum output bandwidth that a NIC is allowed to produce as an integer in Gbit/s. Currently, this must be a number between 1 and 200, allowing you to model NICs between 1 and 200 Gbit/s.
5.1.3.6. profile_interval
¶
The simulation driver periodically samples performance counters in FASED timing model instances and dumps the result to a file on the host.
profile_interval
defines the number of target cycles between samples; setting this field to -1 disables polling.
5.1.3.7. default_hw_config
¶
This sets the server configuration launched by default in the above topologies.
Heterogeneous configurations can be achieved by manually specifying different
names within the topology itself, but all the example_Nconfig
configurations
are homogeneous and use this value for all nodes.
You should set this to one of the hardware configurations you have defined already in
config_hwdb.yaml
. You should set this to the NAME (mapping title) of the
hardware configuration from config_hwdb.yaml
, NOT the actual AGFI or bitstream_tar
itself
(NOT something like agfi-XYZ...
).
5.1.4. tracing
¶
This section manages TracerV-based tracing at simulation runtime. For more details, see the Capturing RISC-V Instruction Traces with TracerV page for more details.
5.1.4.1. enable
¶
This turns tracing on, when set to yes
and off when set to no
. See the Enabling Tracing at Runtime.
5.1.4.2. output_format
¶
This sets the output format for TracerV tracing. See the Selecting a Trace Output Format section.
5.1.4.3. selector
, start
, and end
¶
These configure triggering for TracerV. See the Setting a TracerV Trigger section.
5.1.5. autocounter
¶
This section configures AutoCounter. See the AutoCounter: Profiling with Out-of-Band Performance Counter Collection page for more details.
5.1.5.1. read_rate
¶
This sets the rate at which AutoCounters are read. See the AutoCounter Runtime Parameters section for more details.
5.1.6. workload
¶
This section defines the software that will run on the simulated system.
5.1.6.1. workload_name
¶
This selects a workload to run across the set of simulated nodes. A workload consists of a series of jobs that need to be run on simulated nodes (one job per node).
Workload definitions are located in firesim/deploy/workloads/*.json
.
Some sample workloads:
linux-uniform.json
: This runs the default FireSim Linux distro on as many nodes
as you specify when setting the target_config
parameters.
spec17-intrate.json
: This runs SPECint 2017’s rate benchmarks. In this type of
workload, you should launch EXACTLY the correct number of nodes necessary to run the
benchmark. If you specify fewer nodes, the manager will warn that not all jobs were
assigned to a simulation. If you specify too many simulations and not enough
jobs, the manager will not launch the jobs.
Others can be found in the aforementioned directory. For a description of the JSON format, see Defining Custom Workloads.
5.1.6.2. terminate_on_completion
¶
Set this to no
if you want your Run Farm to keep running once the workload
has completed. Set this to yes
if you want your Run Farm to be TERMINATED
after the workload has completed and results have been copied off.
5.1.6.3. suffix_tag
¶
This allows you to append a string to a workload’s output directory name,
useful for differentiating between successive runs of the same workload,
without renaming the entire workload. For example, specifying
suffix_tag: test-v1
with a workload named super-application
will result
in a workload results directory named
results-workload/DATE--TIME-super-application-test-v1/
.
5.1.7. host_debug
¶
5.1.7.1. zero_out_dram
¶
Set this to yes
to zero-out FPGA-attached DRAM before simulation begins.
This process takes 2-5 minutes. In general, this is not required to produce
deterministic simulations on target machines running linux, but should be
enabled if you observe simulation non-determinism.
5.1.7.2. disable_synth_asserts
¶
Set this to yes
to make the simulation ignore synthesized assertions when
they fire. Otherwise, simulation will print the assertion message and terminate
when an assertion fires.
5.2. config_build.yaml
¶
Here is a sample of this configuration file:
# Build-time build design / AGFI configuration for the FireSim Simulation Manager
# See https://docs.fires.im/en/stable/Advanced-Usage/Manager/Manager-Configuration-Files.html for documentation of all of these params.
# this refers to build farms defined in config_build_farm.yaml
build_farm:
# managerinit replace start
base_recipe: build-farm-recipes/aws_ec2.yaml
# Uncomment and add args to override defaults.
# Arg structure should be identical to the args given
# in the base_recipe.
#recipe_arg_overrides:
# <ARG>: <OVERRIDE>
# managerinit replace end
builds_to_run:
# this section references builds defined in config_build_recipes.yaml
# if you add a build here, it will be built when you run buildbitstream
# Unnetworked designs use a three-domain configuration
# Tiles: 1000 MHz
# <Rational Crossing>
# Uncore: 500 MHz
# <Async Crossing>
# DRAM : 1000 MHz
- firesim_rocket_quadcore_no_nic_l2_llc4mb_ddr3
- firesim_boom_singlecore_no_nic_l2_llc4mb_ddr3
# All NIC-based designs use the legacy FireSim frequency selection, with the
# tiles and uncore running at 3.2 GHz to sustain 200Gb theoretical NIC BW
- firesim_supernode_rocket_singlecore_nic_l2_lbp
- firesim_rocket_quadcore_nic_l2_llc4mb_ddr3
- firesim_boom_singlecore_nic_l2_llc4mb_ddr3
# Configs for tutorials
# - firesim_rocket_singlecore_no_nic_l2_lbp
# - firesim_rocket_singlecore_sha3_nic_l2_llc4mb_ddr3
# - firesim_rocket_singlecore_sha3_no_nic_l2_llc4mb_ddr3
# - firesim_rocket_singlecore_sha3_no_nic_l2_llc4mb_ddr3_printf
# - firesim_gemmini_rocket_singlecore_no_nic
# - firesim_gemmini_printf_rocket_singlecore_no_nic
# Configs for Vitis/XRT
# - vitis_firesim_rocket_singlecore_no_nic
# Config for RHSResearch Nitefury II
# - nitefury_firesim_rocket_singlecore_no_nic
# Configs for Xilinx Alveo U250/U280
# - alveo_u250_firesim_rocket_singlecore_no_nic
# - alveo_u250_firesim_gemmini_rocket_singlecore_no_nic
# - alveo_u200_firesim_rocket_singlecore_no_nic
# - alveo_u280_firesim_rocket_singlecore_no_nic
# Config for Xilinx VCU118
# - xilinx_vcu118_firesim_rocket_singlecore_4GB_no_nic
agfis_to_share:
- firesim_rocket_quadcore_nic_l2_llc4mb_ddr3
- firesim_rocket_quadcore_no_nic_l2_llc4mb_ddr3
- firesim_boom_singlecore_no_nic_l2_llc4mb_ddr3
- firesim_boom_singlecore_nic_l2_llc4mb_ddr3
- firesim_supernode_rocket_singlecore_nic_l2_lbp
# Configs for tutorials
# - firesim_rocket_singlecore_no_nic_l2_lbp
# - firesim_rocket_singlecore_sha3_nic_l2_llc4mb_ddr3
# - firesim_rocket_singlecore_sha3_no_nic_l2_llc4mb_ddr3
# - firesim_rocket_singlecore_sha3_no_nic_l2_llc4mb_ddr3_printf
share_with_accounts:
# To share with a specific user:
somebodysname: 123456789012
# To share publicly:
# public: public
Below, we outline each mapping in detail.
5.2.1. build_farm
¶
In this section, you specify the specific build farm configuration that you wish to use to build FPGA bitstreams.
5.2.1.1. base_recipe
¶
The base_recipe
key/value pair specifies the default set of arguments to use for a particular build farm type.
To change the build farm type, a new base_recipe
file must be provided from deploy/build-farm-recipes
.
You are able to override the arguments given by a base_recipe
by adding keys/values to the recipe_arg_overrides
mapping.
See Build Farm Recipes (build-farm-recipes/*) for more details on the potential build farm recipes that can be used.
5.2.1.2. recipe_arg_overrides
¶
This optional mapping of keys/values allows you to override the default arguments provided by the base_recipe
.
This mapping must match the same mapping structure as the args
mapping within the base_recipe
file given.
Overridden arguments override recursively such that all key/values present in the override args replace the default arguments given
by the base_recipe
. In the case of sequences, a overridden sequence completely replaces the corresponding sequence in the default args.
Additionally, it is not possible to change the default build farm type through these overrides.
This must be done by changing the default base_recipe
.
5.2.2. builds_to_run
¶
In this section, you can list as many build entries as you want to run
for a particular call to the buildbitstream
command (see
config_build_recipes.yaml below for how to define a build entry). For
example, if we want to run the builds named awesome_firesim_config
and quad_core_awesome_firesim_config
, we would
write:
builds_to_run:
- awesome_firesim_config
- quad_core_awesome_firesim_config
5.3. config_build_recipes.yaml
¶
Here is a sample of this configuration file:
# Build-time build recipe configuration for the FireSim Simulation Manager
# See https://docs.fires.im/en/stable/Advanced-Usage/Manager/Manager-Configuration-Files.html for documentation of all of these params.
# this file contains sections that describe hardware designs that /can/ be built.
# edit config_build.yaml to actually "turn on" a config to be built when you run
# buildbitstream
###########
# Schema:
###########
# <NAME>:
# DESIGN: <>
# TARGET_CONFIG: <>
# PLATFORM_CONFIG: Config
# deploy_quintuplet: null
# # NOTE: these platform_config_args are for F1 only
# # they should be set to null if using another platform
# platform_config_args:
# fpga_frequency: null
# build_strategy: null
# post_build_hook: null
# metasim_customruntimeconfig: "path to custom runtime config for metasims"
# bit_builder_recipe:
# # OPTIONAL: overrides for bit builder recipe
# # Arg structure should be identical to the args given
# # in the base_recipe.
# #bit_builder_arg_overrides:
# # <ARG>: <OVERRIDE>
# Quad-core, Rocket-based recipes
# REQUIRED FOR TUTORIALS
firesim_rocket_quadcore_nic_l2_llc4mb_ddr3:
PLATFORM: f1
TARGET_PROJECT: firesim
DESIGN: FireSim
TARGET_CONFIG: WithNIC_DDR3FRFCFSLLC4MB_WithDefaultFireSimBridges_WithFireSimHighPerfConfigTweaks_chipyard.QuadRocketConfig
PLATFORM_CONFIG: WithAutoILA_BaseF1Config
deploy_quintuplet: null
platform_config_args:
fpga_frequency: 90
build_strategy: TIMING
post_build_hook: null
metasim_customruntimeconfig: null
bit_builder_recipe: bit-builder-recipes/f1.yaml
# NB: This has a faster host-clock frequency than the NIC-based design, because
# its uncore runs at half rate relative to the tile.
firesim_rocket_quadcore_no_nic_l2_llc4mb_ddr3:
PLATFORM: f1
TARGET_PROJECT: firesim
DESIGN: FireSim
TARGET_CONFIG: DDR3FRFCFSLLC4MB_WithDefaultFireSimBridges_WithFireSimTestChipConfigTweaks_chipyard.QuadRocketConfig
PLATFORM_CONFIG: WithAutoILA_BaseF1Config
deploy_quintuplet: null
platform_config_args:
fpga_frequency: 140
build_strategy: TIMING
post_build_hook: null
metasim_customruntimeconfig: null
bit_builder_recipe: bit-builder-recipes/f1.yaml
# Single-core, BOOM-based recipes
# REQUIRED FOR TUTORIALS
firesim_boom_singlecore_nic_l2_llc4mb_ddr3:
PLATFORM: f1
TARGET_PROJECT: firesim
DESIGN: FireSim
TARGET_CONFIG: WithNIC_DDR3FRFCFSLLC4MB_WithDefaultFireSimBridges_WithFireSimHighPerfConfigTweaks_chipyard.LargeBoomConfig
PLATFORM_CONFIG: WithAutoILA_BaseF1Config
deploy_quintuplet: null
platform_config_args:
fpga_frequency: 65
build_strategy: TIMING
post_build_hook: null
metasim_customruntimeconfig: null
bit_builder_recipe: bit-builder-recipes/f1.yaml
# NB: This has a faster host-clock frequency than the NIC-based design, because
# its uncore runs at half rate relative to the tile.
firesim_boom_singlecore_no_nic_l2_llc4mb_ddr3:
PLATFORM: f1
TARGET_PROJECT: firesim
DESIGN: FireSim
TARGET_CONFIG: DDR3FRFCFSLLC4MB_WithDefaultFireSimBridges_WithFireSimTestChipConfigTweaks_chipyard.LargeBoomConfig
PLATFORM_CONFIG: WithAutoILA_BaseF1Config
deploy_quintuplet: null
platform_config_args:
fpga_frequency: 65
build_strategy: TIMING
post_build_hook: null
metasim_customruntimeconfig: null
bit_builder_recipe: bit-builder-recipes/f1.yaml
# Single-core, CVA6-based recipes
firesim_cva6_singlecore_no_nic_l2_llc4mb_ddr3:
PLATFORM: f1
TARGET_PROJECT: firesim
DESIGN: FireSim
TARGET_CONFIG: DDR3FRFCFSLLC4MB_WithDefaultFireSimBridges_WithFireSimConfigTweaks_chipyard.CVA6Config
PLATFORM_CONFIG: WithAutoILA_BaseF1Config
deploy_quintuplet: null
platform_config_args:
fpga_frequency: 90
build_strategy: TIMING
post_build_hook: null
metasim_customruntimeconfig: null
bit_builder_recipe: bit-builder-recipes/f1.yaml
# Single-core, Rocket-based recipes with Gemmini
firesim_rocket_singlecore_gemmini_no_nic_l2_llc4mb_ddr3:
PLATFORM: f1
TARGET_PROJECT: firesim
DESIGN: FireSim
TARGET_CONFIG: DDR3FRFCFSLLC4MB_WithDefaultFireSimBridges_WithFireSimConfigTweaks_chipyard.GemminiRocketConfig
PLATFORM_CONFIG: WithAutoILA_BaseF1Config
deploy_quintuplet: null
platform_config_args:
fpga_frequency: 110
build_strategy: TIMING
post_build_hook: null
metasim_customruntimeconfig: null
bit_builder_recipe: bit-builder-recipes/f1.yaml
# RAM Optimizations enabled by adding _MCRams PLATFORM_CONFIG string
firesim_boom_singlecore_no_nic_l2_llc4mb_ddr3_ramopts:
PLATFORM: f1
TARGET_PROJECT: firesim
DESIGN: FireSim
TARGET_CONFIG: DDR3FRFCFSLLC4MB_WithDefaultFireSimBridges_WithFireSimTestChipConfigTweaks_chipyard.LargeBoomConfig
PLATFORM_CONFIG: WithAutoILA_MCRams_BaseF1Config
deploy_quintuplet: null
platform_config_args:
fpga_frequency: 90
build_strategy: TIMING
post_build_hook: null
metasim_customruntimeconfig: null
bit_builder_recipe: bit-builder-recipes/f1.yaml
# Supernode configurations -- multiple instances of an SoC in a single simulator
firesim_supernode_rocket_singlecore_nic_l2_lbp:
PLATFORM: f1
TARGET_PROJECT: firesim
DESIGN: FireSim
TARGET_CONFIG: WithNIC_SupernodeFireSimRocketConfig
PLATFORM_CONFIG: WithAutoILA_BaseF1Config
deploy_quintuplet: null
platform_config_args:
fpga_frequency: 85
build_strategy: TIMING
post_build_hook: null
metasim_customruntimeconfig: null
bit_builder_recipe: bit-builder-recipes/f1.yaml
# MIDAS Examples -- BUILD SUPPORT ONLY; Can't launch driver correctly on run farm
midasexamples_gcd:
PLATFORM: f1
TARGET_PROJECT: midasexamples
DESIGN: GCD
TARGET_CONFIG: NoConfig
PLATFORM_CONFIG: DefaultF1Config
deploy_quintuplet: null
platform_config_args:
fpga_frequency: 75
build_strategy: TIMING
post_build_hook: null
metasim_customruntimeconfig: null
bit_builder_recipe: bit-builder-recipes/f1.yaml
# Additional Tutorial Config
firesim_rocket_singlecore_no_nic_l2_lbp:
PLATFORM: f1
TARGET_PROJECT: firesim
DESIGN: FireSim
TARGET_CONFIG: WithDefaultFireSimBridges_WithFireSimHighPerfConfigTweaks_chipyard.RocketConfig
PLATFORM_CONFIG: BaseF1Config
deploy_quintuplet: null
platform_config_args:
fpga_frequency: 90
build_strategy: TIMING
post_build_hook: null
metasim_customruntimeconfig: null
bit_builder_recipe: bit-builder-recipes/f1.yaml
# Additional Tutorial Config
firesim_rocket_singlecore_sha3_nic_l2_llc4mb_ddr3:
PLATFORM: f1
TARGET_PROJECT: firesim
DESIGN: FireSim
TARGET_CONFIG: WithNIC_DDR3FRFCFSLLC4MB_WithDefaultFireSimBridges_WithFireSimHighPerfConfigTweaks_chipyard.Sha3RocketConfig
PLATFORM_CONFIG: BaseF1Config
deploy_quintuplet: null
platform_config_args:
fpga_frequency: 65
build_strategy: TIMING
post_build_hook: null
metasim_customruntimeconfig: null
bit_builder_recipe: bit-builder-recipes/f1.yaml
# Additional Tutorial Config
firesim_rocket_singlecore_sha3_no_nic_l2_llc4mb_ddr3:
PLATFORM: f1
TARGET_PROJECT: firesim
DESIGN: FireSim
TARGET_CONFIG: DDR3FRFCFSLLC4MB_WithDefaultFireSimBridges_WithFireSimHighPerfConfigTweaks_chipyard.Sha3RocketConfig
PLATFORM_CONFIG: BaseF1Config
deploy_quintuplet: null
platform_config_args:
fpga_frequency: 65
build_strategy: TIMING
post_build_hook: null
metasim_customruntimeconfig: null
bit_builder_recipe: bit-builder-recipes/f1.yaml
# Additional Tutorial Config
firesim_rocket_singlecore_sha3_no_nic_l2_llc4mb_ddr3_printf:
PLATFORM: f1
TARGET_PROJECT: firesim
DESIGN: FireSim
TARGET_CONFIG: DDR3FRFCFSLLC4MB_WithDefaultFireSimBridges_WithFireSimHighPerfConfigTweaks_chipyard.Sha3RocketPrintfConfig
PLATFORM_CONFIG: WithPrintfSynthesis_BaseF1Config
deploy_quintuplet: null
platform_config_args:
fpga_frequency: 30
build_strategy: TIMING
post_build_hook: null
metasim_customruntimeconfig: null
bit_builder_recipe: bit-builder-recipes/f1.yaml
# Additional Xilinx Vitis/XRT-only Config
vitis_firesim_rocket_singlecore_no_nic:
PLATFORM: vitis
TARGET_PROJECT: firesim
DESIGN: FireSim
TARGET_CONFIG: FireSimRocketMMIOOnlyConfig
PLATFORM_CONFIG: BaseVitisConfig
deploy_quintuplet: null
platform_config_args:
fpga_frequency: 30 # previously reached 140. using 30 for build speed
build_strategy: TIMING
post_build_hook: null
metasim_customruntimeconfig: null
bit_builder_recipe: bit-builder-recipes/vitis.yaml
# Additional Tutorial Config
firesim_gemmini_rocket_singlecore_no_nic:
PLATFORM: f1
TARGET_PROJECT: firesim
DESIGN: FireSim
TARGET_CONFIG: FireSimLeanGemminiRocketConfig
PLATFORM_CONFIG: BaseF1Config
deploy_quintuplet: null
platform_config_args:
fpga_frequency: 30 # AJG: conservative for now, later sweep for higher frequency
build_strategy: TIMING
post_build_hook: null
metasim_customruntimeconfig: null
bit_builder_recipe: bit-builder-recipes/f1.yaml
# Additional Tutorial Config
firesim_gemmini_printf_rocket_singlecore_no_nic:
PLATFORM: f1
TARGET_PROJECT: firesim
DESIGN: FireSim
TARGET_CONFIG: FireSimLeanGemminiPrintfRocketConfig
PLATFORM_CONFIG: WithPrintfSynthesis_BaseF1Config
deploy_quintuplet: null
platform_config_args:
fpga_frequency: 10 # AJG: conservative for now, later sweep for higher frequency
build_strategy: TIMING
post_build_hook: null
metasim_customruntimeconfig: null
bit_builder_recipe: bit-builder-recipes/f1.yaml
# Additional Xilinx Alveo U250-only Config
alveo_u250_firesim_rocket_singlecore_no_nic:
PLATFORM: xilinx_alveo_u250
TARGET_PROJECT: firesim
DESIGN: FireSim
TARGET_CONFIG: FireSimRocketConfig
PLATFORM_CONFIG: BaseXilinxAlveoConfig
deploy_quintuplet: null
platform_config_args:
fpga_frequency: 60
build_strategy: TIMING
post_build_hook: null
metasim_customruntimeconfig: null
bit_builder_recipe: bit-builder-recipes/xilinx_alveo_u250.yaml
# Additional Tutorial Config
# Additional Xilinx Alveo U250-only Config
alveo_u250_firesim_gemmini_rocket_singlecore_no_nic:
PLATFORM: xilinx_alveo_u250
TARGET_PROJECT: firesim
DESIGN: FireSim
TARGET_CONFIG: FireSimLeanGemminiRocketConfig
PLATFORM_CONFIG: BaseXilinxAlveoConfig
deploy_quintuplet: null
platform_config_args:
fpga_frequency: 60
build_strategy: TIMING
post_build_hook: null
metasim_customruntimeconfig: null
bit_builder_recipe: bit-builder-recipes/xilinx_alveo_u250.yaml
# Additional Xilinx Alveo U280-only Config
alveo_u280_firesim_rocket_singlecore_no_nic:
PLATFORM: xilinx_alveo_u280
TARGET_PROJECT: firesim
DESIGN: FireSim
TARGET_CONFIG: FireSimRocketConfig
PLATFORM_CONFIG: BaseXilinxAlveoConfig
deploy_quintuplet: null
platform_config_args:
fpga_frequency: 60
build_strategy: TIMING
post_build_hook: null
metasim_customruntimeconfig: null
bit_builder_recipe: bit-builder-recipes/xilinx_alveo_u280.yaml
# Additional Xilinx Alveo U200-only Config
alveo_u200_firesim_rocket_singlecore_no_nic:
PLATFORM: xilinx_alveo_u200
TARGET_PROJECT: firesim
DESIGN: FireSim
TARGET_CONFIG: FireSimRocketConfig
PLATFORM_CONFIG: BaseXilinxAlveoConfig
deploy_quintuplet: null
platform_config_args:
fpga_frequency: 60
build_strategy: TIMING
post_build_hook: null
metasim_customruntimeconfig: null
bit_builder_recipe: bit-builder-recipes/xilinx_alveo_u200.yaml
# Xilinx VCU118-only Config
xilinx_vcu118_firesim_rocket_singlecore_4GB_no_nic:
PLATFORM: xilinx_vcu118
TARGET_PROJECT: firesim
DESIGN: FireSim
TARGET_CONFIG: FireSimRocket4GiBDRAMConfig
PLATFORM_CONFIG: BaseXilinxVCU118Config
deploy_quintuplet: null
platform_config_args:
fpga_frequency: 90
build_strategy: TIMING
post_build_hook: null
metasim_customruntimeconfig: null
bit_builder_recipe: bit-builder-recipes/xilinx_vcu118.yaml
# RHSResearch Nitefury II-only Config
nitefury_firesim_rocket_singlecore_no_nic:
PLATFORM: rhsresearch_nitefury_ii
TARGET_PROJECT: firesim
DESIGN: FireSim
TARGET_CONFIG: FireSimRocket1GiBDRAMConfig
PLATFORM_CONFIG: BaseNitefuryConfig
deploy_quintuplet: null
platform_config_args:
fpga_frequency: 50
build_strategy: TIMING
post_build_hook: null
metasim_customruntimeconfig: null
bit_builder_recipe: bit-builder-recipes/rhsresearch_nitefury_ii.yaml
Below, we outline each section and parameter in detail.
5.3.1. Build definition sections, e.g. awesome_firesim_config
¶
In this file, you can specify as many build definition sections as you want,
each with a header like awesome_firesim_config
(i.e. a nice, short name
you made up). Such a section must contain the following fields:
5.3.1.1. DESIGN
¶
This specifies the basic target design that will be built. Unless you
are defining a custom system, this should be set to FireSim
.
We describe this in greater detail in Generating Different
Targets).
5.3.1.2. TARGET_CONFIG
¶
This specifies the hardware configuration of the target being simulated. Some
examples include FireSimRocketConfig
and FireSimQuadRocketConfig
.
We describe this in greater detail in Generating Different
Targets).
5.3.1.3. PLATFORM_CONFIG
¶
This specifies parameters to pass to the compiler (Golden Gate). Notably, PLATFORM_CONFIG can be used to enable debugging tools like assertion synthesis, and resource optimizations like instance multithreading. Critically, it also calls out the host-platform (e.g., F1) to compile against: this defines the widths of internal simulation interfaces and specifies resource limits (e.g., how much DRAM is available on the platform).
5.3.1.3.1. platform_build_args
¶
These configure the bitstream build, and are host-platform-agnostic. Platform-specific arguments, like the Vitis platform (“DEVICE”), are captured as arguments to the bitbuilder.
5.3.1.3.1.1. fpga_frequency
¶
Specifies the host FPGA frequency for a bitstream build.
5.3.1.3.1.2. build_strategy
¶
Specifies a pre-canned set of strategies and directives to pass to the bitstream build. Note, these are implemented differently on different host platforms, but try to optimize for the same things. Strategies supported across both Vitis, Xilinx Alveo U250/U280, and EC2 F1 include:
TIMING
: Optimize for improved fmax.
AREA
: Optimize for reduced resource utilization.
Names are derived AWS’s strategy set.
5.3.1.4. TARGET_PROJECT
(Optional)¶
This specifies the target project in which the target is defined (this is described
in greater detail here). If
TARGET_PROJECT
is undefined the manager will default to firesim
.
Setting TARGET_PROJECT
is required for building the MIDAS examples
(TARGET_PROJECT: midasexamples
) with the manager, or for building a
user-provided target project.
5.3.1.5. PLATFORM
(Optional)¶
This specifies the platform for which the target will be built for (this is described
in greater detail here). If
PLATFORM
is undefined the manager will default to f1
.
5.3.1.6. deploy_quintuplet
¶
This allows you to override the deployquintuplet
stored with the AGFI.
Otherwise, the PLATFORM
/TARGET_PROJECT
/DESIGN
/TARGET_CONFIG
/PLATFORM_CONFIG
you specify
above will be used. See the AGFI Tagging section for more details. Most likely,
you should leave this set to null
. This is usually only used if you have
proprietary RTL that you bake into an FPGA image, but don’t want to share with
users of the simulator.
5.3.1.7. post_build_hook
¶
(Optional) Provide an a script to run on the results copied back from a _single_ build instance. Upon completion of each design’s build, the manager invokes this script and passing the absolute path to that instance’s build-results directory as it’s first argument.
5.3.1.8. metasim_customruntimeconfig
¶
This is an advanced feature - under normal conditions, you can use the default
parameters generated automatically by the simulator by setting this field to
null
for metasimulations. If you want to customize runtime parameters for certain parts of
the metasimulation (e.g. the DRAM model’s runtime parameters), you can place
a custom config file in sim/custom-runtime-configs/
. Then, set this field
to the relative name of the config. For example,
sim/custom-runtime-configs/GREATCONFIG.conf
becomes
metasim_customruntimeconfig: GREATCONFIG.conf
.
5.3.1.9. bit_builder_recipe
¶
This specifies the bitstream type to generate for a particular recipe.
This must point to a file in deploy/bit-builder-recipes/
.
See Bit Builder Recipes (bit-builder-recipes/*) for more details on bit builders and their arguments.
5.3.1.10. bit_builder_arg_overrides
¶
This optional mapping of keys/values allows you to override the default arguments provided by the bit_builder_recipe
.
This mapping must match the same mapping structure as the args
mapping within the bit_builder_recipe
file given.
Overridden arguments override recursively such that all key/values present in the override args replace the default arguments given
by the bit_builder_recipe
. In the case of sequences, a overridden sequence completely replaces the corresponding sequence in the default args.
Additionally, it is not possible to change the default bit builder type through these overrides.
This must be done by changing the default bit_builder_recipe
.
5.4. config_hwdb.yaml
¶
Here is a sample of this configuration file:
# Hardware config database for FireSim Simulation Manager
# See https://docs.fires.im/en/stable/Advanced-Usage/Manager/Manager-Configuration-Files.html for documentation of all of these params.
# Hardware configs represent a combination of an agfi, a deployquintuplet override
# (if needed), and a custom runtime config (if needed)
# The AGFIs provided below are public and available to all users.
# Only AGFIs for the latest release of FireSim are guaranteed to be available.
# If you are using an older version of FireSim, you will need to generate your
# own images.
# DOCREF START: Example HWDB Entry
firesim_boom_singlecore_nic_l2_llc4mb_ddr3:
agfi: agfi-00e99bd64c0643ac9
deploy_quintuplet_override: null
custom_runtime_config: null
# DOCREF END: Example HWDB Entry
firesim_boom_singlecore_no_nic_l2_llc4mb_ddr3:
agfi: agfi-0d475d004a5929ce1
deploy_quintuplet_override: null
custom_runtime_config: null
firesim_gemmini_printf_rocket_singlecore_no_nic:
agfi: agfi-0ace16d35c5758893
deploy_quintuplet_override: null
custom_runtime_config: null
firesim_gemmini_rocket_singlecore_no_nic:
agfi: agfi-05eec5fb565f7cfa3
deploy_quintuplet_override: null
custom_runtime_config: null
firesim_rocket_quadcore_nic_l2_llc4mb_ddr3:
agfi: agfi-03c40fa49ed5c84e8
deploy_quintuplet_override: null
custom_runtime_config: null
firesim_rocket_quadcore_no_nic_l2_llc4mb_ddr3:
agfi: agfi-0e55df16a9ca937aa
deploy_quintuplet_override: null
custom_runtime_config: null
firesim_rocket_singlecore_no_nic_l2_lbp:
agfi: agfi-0cfc97258aa8389f0
deploy_quintuplet_override: null
custom_runtime_config: null
firesim_rocket_singlecore_sha3_nic_l2_llc4mb_ddr3:
agfi: agfi-02e4056f9bec5a240
deploy_quintuplet_override: null
custom_runtime_config: null
firesim_rocket_singlecore_sha3_no_nic_l2_llc4mb_ddr3:
agfi: agfi-0d8abef077c23a4de
deploy_quintuplet_override: null
custom_runtime_config: null
firesim_rocket_singlecore_sha3_no_nic_l2_llc4mb_ddr3_printf:
agfi: agfi-033e840230f51668f
deploy_quintuplet_override: null
custom_runtime_config: null
firesim_supernode_rocket_singlecore_nic_l2_lbp:
agfi: agfi-049822fd73bc2fca1
deploy_quintuplet_override: null
custom_runtime_config: null
vitis_firesim_rocket_singlecore_no_nic:
bitstream_tar: https://raw.githubusercontent.com/firesim/firesim-public-bitstreams/9697ea5f5f297e386c5c6f60ba5a71ee3020f93f/vitis/vitis_firesim_rocket_singlecore_no_nic.tar.gz
deploy_quintuplet_override: null
custom_runtime_config: null
vitis_firesim_gemmini_rocket_singlecore_no_nic:
bitstream_tar: https://raw.githubusercontent.com/firesim/firesim-public-bitstreams/0af81b912264abbe3f90f8140987814291090560/vitis/vitis_firesim_gemmini_rocket_singlecore_no_nic.tar.gz
deploy_quintuplet_override: null
custom_runtime_config: null
alveo_u250_firesim_rocket_singlecore_no_nic:
bitstream_tar: https://raw.githubusercontent.com/firesim/firesim-public-bitstreams/ac2cd0c123e1755a666d48709021d29abb440ab6/xilinx_alveo_u250/alveo_u250_firesim_rocket_singlecore_no_nic.tar.gz
deploy_quintuplet_override: null
custom_runtime_config: null
alveo_u250_firesim_gemmini_rocket_singlecore_no_nic:
bitstream_tar: https://raw.githubusercontent.com/firesim/firesim-public-bitstreams/ebeffff3cd54225fbbcac36a8f2ced5645f2079e/xilinx_alveo_u250/alveo_u250_firesim_gemmini_rocket_singlecore_no_nic.tar.gz
deploy_quintuplet_override: null
custom_runtime_config: null
alveo_u200_firesim_rocket_singlecore_no_nic:
bitstream_tar: https://raw.githubusercontent.com/firesim/firesim-public-bitstreams/9fa5d271407bd7057885c02f19f510842f3f3c95/xilinx_alveo_u200/alveo_u200_firesim_rocket_singlecore_no_nic.tar.gz
deploy_quintuplet_override: null
custom_runtime_config: null
alveo_u280_firesim_rocket_singlecore_no_nic:
bitstream_tar: https://raw.githubusercontent.com/firesim/firesim-public-bitstreams/808bce6788100b700730bd5b39b86e5c004cf43d/xilinx_alveo_u280/alveo_u280_firesim_rocket_singlecore_no_nic.tar.gz
deploy_quintuplet_override: null
custom_runtime_config: null
xilinx_vcu118_firesim_rocket_singlecore_4GB_no_nic:
bitstream_tar: https://raw.githubusercontent.com/firesim/firesim-public-bitstreams/c3adc1471a61cb1415a0c800271baad5bf753d02/xilinx_vcu118/xilinx_vcu118_firesim_rocket_singlecore_4GB_no_nic.tar.gz
deploy_quintuplet_override: null
custom_runtime_config: null
nitefury_firesim_rocket_singlecore_no_nic:
bitstream_tar: https://raw.githubusercontent.com/firesim/firesim-public-bitstreams/7fa19e457374d98cc7abf7a196b93db23db2dc52/rhsresearch_nitefury_ii/nitefury_firesim_rocket_singlecore_no_nic.tar.gz
deploy_quintuplet_override: null
custom_runtime_config: null
This file tracks hardware configurations that you can deploy as simulated nodes
in FireSim. Each such configuration contains a name for easy reference in higher-level
configurations, defined in the section header, an handle to a bitstream (i.e. an AGFI or bitstream_tar
path), which represents the
FPGA image, a custom runtime config, if one is needed, and a deploy quintuplet
override if one is necessary.
When you build a new bitstream, you should put it in this file so that it can be referenced from your other configuration files.
The following is an example section from this file - you can add as many of these as necessary:
firesim_boom_singlecore_nic_l2_llc4mb_ddr3:
agfi: agfi-00e99bd64c0643ac9
deploy_quintuplet_override: null
custom_runtime_config: null
Here are the components of these entries:
5.4.1. The name: firesim_boom_singlecore_nic_l2_llc4mb_ddr3
¶
In this example, firesim_boom_singlecore_nic_l2_llc4mb_ddr3
is the name that will be
used to reference this hardware design in other configuration locations. The following
items describe this hardware configuration:
5.4.1.1. agfi
¶
This represents the AGFI (FPGA Image) used by this hardware configuration.
Only used in AWS EC2 F1 FireSim configurations (a bitstream_tar
key/value cannot exist with this
key/value in the same recipe).
5.4.1.2. bitstream_tar
¶
This is not shown in the example entry above, but would be used for an on-premises bitstream.
Indicates where the bitstream (FPGA Image) and metadata associated with it is located, may be one of:
A Uniform Resource Identifier (URI), (see Manager URI Paths for details)
A filesystem path available to the manager. Local paths are relative to the deploy folder.
5.4.1.3. deploy_quintuplet_override
¶
This is an advanced feature - under normal conditions, you should leave this set to null
, so that the
manager uses the configuration quintuplet that is automatically stored with the
bitstream metadata at build time. Advanced users can set this to a different
value to build and use a different driver when deploying simulations. Since
the driver depends on logic now hardwired into the
FPGA bitstream, drivers cannot generally be changed without requiring FPGA
recompilation.
5.4.1.4. custom_runtime_config
¶
This is an advanced feature - under normal conditions, you can use the default
parameters generated automatically by the simulator by setting this field to
null
. If you want to customize runtime parameters for certain parts of
the simulation (e.g. the DRAM model’s runtime parameters), you can place
a custom config file in sim/custom-runtime-configs/
. Then, set this field
to the relative name of the config. For example,
sim/custom-runtime-configs/GREATCONFIG.conf
becomes
custom_runtime_config: GREATCONFIG.conf
.
5.4.1.5. driver_tar
¶
The value for this key can be one of:
A Uniform Resource Identifier (URI), (see Manager URI Paths for details)
A filesystem path available to the manager. Local paths are relative to the deploy folder.
When this key is present, the FireSim FPGA-driver software will not be built from source. Instead, during firesim infrasetup, this file will be deployed and extracted into the sim_slot_X folder on the run farm instance. This file may be a .tar, .tar.gz, .tar.bz2 or any other format that GNU tar (version 1.26) can automatically detect. The purpose of this feature is to enable advanced CI configurations where the driver build step is decoupled. For now this can only accept a path to a file on the manager’s local filesystem. This key can also be a URI.
5.4.2. Add more hardware config sections, like NAME_GOES_HERE_2
¶
You can add as many of these entries to config_hwdb.yaml
as you want, following the format
discussed above (i.e. you provide agfi
or bitstream_tar
, deploy_quintuplet_override
, and custom_runtime_config
).
5.5. Run Farm Recipes (run-farm-recipes/*
)¶
Here is an example of this configuration file:
# AWS EC2 run farm hosts recipe.
# all fields are required but can be overridden in the `*_runtime.yaml`
run_farm_type: AWSEC2F1
args:
# managerinit arg start
# tag to apply to run farm hosts
run_farm_tag: mainrunfarm
# enable expanding run farm by run_farm_hosts given
always_expand_run_farm: true
# minutes to retry attempting to request instances
launch_instances_timeout_minutes: 60
# run farm host market to use (ondemand, spot)
run_instance_market: ondemand
# if using spot instances, determine the interrupt behavior (terminate, stop, hibernate)
spot_interruption_behavior: terminate
# if using spot instances, determine the max price
spot_max_price: ondemand
# default location of the simulation directory on the run farm host
default_simulation_dir: /home/centos
# run farm hosts to spawn: a mapping from a spec below (which is an EC2
# instance type) to the number of instances of the given type that you
# want in your runfarm.
run_farm_hosts_to_use:
- f1.16xlarge: 0
- f1.4xlarge: 0
- f1.2xlarge: 1
- m4.16xlarge: 0
- z1d.3xlarge: 0
- z1d.6xlarge: 0
- z1d.12xlarge: 0
# managerinit arg end
# REQUIRED: List of host "specifications", i.e. re-usable collections of
# host parameters.
#
# On EC2, most users will never need to edit this section,
# unless you want to add new host instance types.
#
# The "name" of a spec below (e.g. "f1.2xlarge" below) MUST be a valid EC2
# instance type and is used to refer to the spec above.
#
# Besides required parameters shown below, each can have multiple OPTIONAL
# arguments, called "override_*", corresponding to the "default_*" arguments
# specified above. Each "override_*" argument overrides the corresponding
# "default_*" argument in that run host spec.
#
# Optional params include:
# override_simulation_dir: /scratch/specific-build-host-build-dir
# override_platform: EC2InstanceDeployManager
run_farm_host_specs:
- f1.2xlarge: # REQUIRED: On EC2, the spec name MUST be an EC2 instance type.
# REQUIRED: number of FPGAs on the machine
num_fpgas: 1
# REQUIRED: number of metasims this machine can host
num_metasims: 0
# REQUIRED: whether it is acceptable to use machines of this spec
# to host ONLY switches (e.g. any attached FPGAs are "wasted")
use_for_switch_only: false
- f1.4xlarge:
num_fpgas: 2
num_metasims: 0
use_for_switch_only: false
- f1.16xlarge:
num_fpgas: 8
num_metasims: 0
use_for_switch_only: false
- m4.16xlarge:
num_fpgas: 0
num_metasims: 0
use_for_switch_only: true
- z1d.3xlarge:
num_fpgas: 0
num_metasims: 1
use_for_switch_only: false
- z1d.6xlarge:
num_fpgas: 0
num_metasims: 2
use_for_switch_only: false
- z1d.12xlarge:
num_fpgas: 0
num_metasims: 8
use_for_switch_only: false
5.5.1. run_farm_type
¶
This key/value specifies a run farm class to use for launching, managing, and terminating
run farm hosts used for simulations.
By default, run farm classes can be found in deploy/runtools/run_farm.py. However, you can specify
your own custom run farm classes by adding your python file to the PYTHONPATH
.
For example, to use the AWSEC2F1
run farm class, you would write run_farm_type: AWSEC2F1
.
5.5.2. args
¶
This section specifies all arguments needed for the specific run_farm_type
used.
For a list of arguments needed for a run farm class, users should refer to
the _parse_args
function in the run farm class given by run_farm_type
.
5.5.3. aws_ec2.yaml
run farm recipe¶
The run farm recipe shown above configures a FireSim run farm to use AWS EC2 instances. It contains several key/value pairs:
5.5.3.1. run_farm_tag
¶
Use run_farm_tag
to differentiate between different Run Farms in FireSim.
Having multiple config_runtime.yaml
files with different run_farm_tag
values allows you to run many experiments at once from the same manager instance.
The instances launched by the launchrunfarm
command will be tagged with
this value. All later operations done by the manager rely on this tag, so
you should not change it unless you are done with your current Run Farm.
Per AWS restrictions, this tag can be no longer than 255 characters.
5.5.3.2. always_expand_run_farm
¶
When true
(the default behavior when not given) the number of instances
of each type (see f1.16xlarges
etc. below) are launched every time you
run launchrunfarm
.
When false
, launchrunfarm
looks for already existing instances that
match run_farm_tag
and treat f1.16xlarges
(and other ‘instance-type’
values below) as a total count.
For example, if you have f1.2xlarges
set to 100 and the first time you
run launchrunfarm
you have launch_instances_timeout_minutes
set to 0
(i.e. giveup after receiving a ClientError
for each AvailabilityZone) and
AWS is only able to provide you 75 f1.2xlarges
because of capacity issues,
always_expand_runfarm
changes the behavior of launchrunfarm
in subsequent
attempts. yes
means launchrunfarm
will try to launch 100 f1.2xlarges
again. no
means that launchrunfarm
will only try to launch an additional
25 f1.2xlarges
because it will see that there are already 75 that have been launched
with the same run_farm_tag
.
5.5.3.3. launch_instances_timeout_minutes
¶
Integer number of minutes that the launchrunfarm
command will attempt to
request new instances before giving up. This limit is used for each of the types
of instances being requested. For example, if you set to 60,
and you are requesting all four types of instances, launchrunfarm
will try
to launch each instance type for 60 minutes, possibly trying up to a total of
four hours.
This limit starts to be applied from the first time launchrunfarm
receives a
ClientError
response in all AvailabilityZones (AZs) for your region. In other words,
if you request more instances than can possibly be requested in the given limit but AWS
is able to satisfy all of the requests, the limit will not be enforced.
To experience the old (<= 1.12) behavior, set this limit to 0 and launchrunfarm
will exit the first time it receives ClientError
across all AZ’s. The old behavior
is also the default if launch_instances_timeout_minutes
is not included.
5.5.3.4. run_instance_market
¶
You can specify either spot
or ondemand
here, to use one of those
markets on AWS.
5.5.3.5. spot_interruption_behavior
¶
When run_instance_market: spot
, this value determines what happens to an instance
if it receives the interruption signal from AWS. You can specify either
hibernate
, stop
, or terminate
.
5.5.3.6. spot_max_price
¶
When run_instance_market: spot
, this value determines the max price you are
willing to pay per instance, in dollars. You can also set it to ondemand
to set your max to the on-demand price for the instance.
5.5.3.7. default_simulation_dir
¶
This is the path on the run farm host that simulations will run out of.
5.5.3.8. run_farm_hosts_to_use
¶
This is a sequence of unique specifications (given by run_farm_host_specs
) to number of instances needed.
Set these key/value pairs respectively based on the number and types of instances
you need. While we could automate this setting, we choose not to, so that
users are never surprised by how many instances they are running.
Note that these values are ONLY used to launch instances. After launch, the
manager will query the AWS API to find the instances of each type that have the
run_farm_tag
set above assigned to them.
Also refer to always_expand_runfarm
which determines whether launchrunfarm
treats these counts as an incremental amount to be launched every time it is envoked
or a total number of instances of that type and run_farm_tag
that should be made
to exist. Note, launchrunfarm
will never terminate instances.
5.5.3.9. run_farm_host_specs
¶
This is a sequence of specifications that describe a AWS EC2 instance and its properties.
A sequence consists of the AWS EC2 instance name (i.e. f1.2xlarge
) and number of FPGAs it supports
(num_fpgas
), number of metasims it could support (num_metasims
), and if the instance
should only host switch simulations (use_for_switch_only
). Additionally, a specification can optionally add
override_simulation_dir
to override the default_simulation_dir
for that specific run farm host.
Similarly, a specification can optionally add override_platform
to choose a different default deploy manager platform for
that specific run farm host (for more details on this see the following section). By default, the deploy manager is setup
for AWS EC2 simulations.
5.5.4. externally_provisioned.yaml
run farm recipe¶
This run farm allows users to provide a list of pre-setup unmanaged run farm hosts (by hostname or IP address) that they can run simulations on. Note that this run farm type does not launch or terminate the run farm hosts. This functionality should be handled by the user. For example, users can use this run farm type to run simulations locally.
Here is an example of this configuration file:
# Unmanaged list of run farm hosts. Assumed that they are pre-setup to run simulations.
# all fields are required but can be overridden in the `*_runtime.yaml`
run_farm_type: ExternallyProvisioned
args:
# managerinit arg start
# REQUIRED: default platform used for run farm hosts. this is a class specifying
# how to run simulations on a run farm host.
default_platform: EC2InstanceDeployManager
# REQUIRED: default directory where simulations are run out of on the run farm hosts
default_simulation_dir: /home/centos
# REQUIRED: List of unique hostnames/IP addresses, each with their
# corresponding specification that describes the properties of the host.
#
# Ex:
# run_farm_hosts_to_use:
# # use localhost which is described by "four_fpgas_spec" below.
# - localhost: four_fpgas_spec
# # supply IP address, which points to a machine that is described
# # by "four_fpgas_spec" below.
# - "111.111.1.111": four_fpgas_spec
run_farm_hosts_to_use:
- localhost: one_fpgas_spec
# managerinit arg end
# REQUIRED: List of host "specifications", i.e. re-usable collections of
# host parameters.
#
# The "name" of a spec (e.g. "four_fpgas_spec" below) is user-determined
# and is used to refer to the spec above.
#
# Besides required parameters shown below, each can have multiple OPTIONAL
# arguments, called "override_*", corresponding to the "default_*" arguments
# specified above. Each "override_*" argument overrides the corresponding
# "default_*" argument in that run host spec.
#
# Optional params include:
# override_simulation_dir: /scratch/specific-build-host-build-dir
# override_platform: EC2InstanceDeployManager
run_farm_host_specs:
- four_fpgas_spec:
# REQUIRED: number of FPGAs on the machine
num_fpgas: 4
# REQUIRED: number of metasims this machine can host
num_metasims: 0
# REQUIRED: whether it is acceptable to use machines of this spec
# to host ONLY switches (e.g. any attached FPGAs are "wasted")
use_for_switch_only: false
- four_metasims_spec:
num_fpgas: 0
num_metasims: 4
use_for_switch_only: false
- switch_only_spec:
num_fpgas: 0
num_metasims: 0
use_for_switch_only: true
- one_fpga_spec:
num_fpgas: 1
num_metasims: 0
use_for_switch_only: false
- one_fpgas_spec:
num_fpgas: 1
num_metasims: 0
use_for_switch_only: false
- two_fpgas_spec:
num_fpgas: 2
num_metasims: 0
use_for_switch_only: false
- three_fpgas_spec:
num_fpgas: 3
num_metasims: 0
use_for_switch_only: false
- four_fpgas_spec:
num_fpgas: 4
num_metasims: 0
use_for_switch_only: false
- five_fpgas_spec:
num_fpgas: 5
num_metasims: 0
use_for_switch_only: false
- six_fpgas_spec:
num_fpgas: 6
num_metasims: 0
use_for_switch_only: false
- seven_fpgas_spec:
num_fpgas: 7
num_metasims: 0
use_for_switch_only: false
- eight_fpgas_spec:
num_fpgas: 8
num_metasims: 0
use_for_switch_only: false
- nine_fpgas_spec:
num_fpgas: 9
num_metasims: 0
use_for_switch_only: false
- ten_fpgas_spec:
num_fpgas: 10
num_metasims: 0
use_for_switch_only: false
- eleven_fpgas_spec:
num_fpgas: 11
num_metasims: 0
use_for_switch_only: false
- twelve_fpgas_spec:
num_fpgas: 12
num_metasims: 0
use_for_switch_only: false
- thirteen_fpgas_spec:
num_fpgas: 13
num_metasims: 0
use_for_switch_only: false
- fourteen_fpgas_spec:
num_fpgas: 14
num_metasims: 0
use_for_switch_only: false
- fifteen_fpgas_spec:
num_fpgas: 15
num_metasims: 0
use_for_switch_only: false
- sixteen_fpgas_spec:
num_fpgas: 16
num_metasims: 0
use_for_switch_only: false
5.5.4.1. default_platform
¶
This key/value specifies a default deploy platform (also known as a deploy manager) class to use for managing
simulations across all run farm hosts.
For example, this class manages how to flash FPGAs with bitstreams, how to copy back results, and how to check if a simulation is running.
By default, deploy platform classes can be found in deploy/runtools/run_farm_deploy_managers.py. However, you can specify
your own custom run farm classes by adding your python file to the PYTHONPATH
.
There are default deploy managers / platforms that correspond to AWS EC2 F1 FPGAs, Vitis FPGAs, Xilinx Alveo U250/U280 FPGAs, Xilinx VCU118 FPGAs, and RHS Research Nitefury II FPGAs: EC2InstanceDeployManager
, VitisInstanceDeployManager
, Xilinx{AlveoU250,AlveoU280,VCU118}InstanceDeployManager
, and RHSResearchNitefuryIIInstanceDeployManager
respectively.
For example, to use the EC2InstanceDeployManager
deploy platform class, you would write default_platform: EC2InstanceDeployManager
.
5.5.4.2. default_simulation_dir
¶
This is the default path on all run farm hosts that simulations will run out of.
5.5.4.3. run_farm_hosts_to_use
¶
This is a sequence of unique hostnames/IP address to specifications (given by run_farm_host_specs
).
Set these key/value pairs respectively to map unmanaged run farm hosts
to their description (given by a specification).
For example, to run simulations locally, a user can write a sequence element with - localhost: four_fpgas_spec
to
indicate that localhost
should be used and that it has a type of four_fpgas_spec
.
5.5.4.4. run_farm_host_specs
¶
This is a sequence of specifications that describe an unmanaged run farm host and its properties.
A sequence consists of the specification name (i.e. four_fpgas_spec
) and number of FPGAs it supports
(num_fpgas
), number of metasims it could support (num_metasims
), and if the instance
should only host switch simulations (use_for_switch_only
). Additionally, a specification can optionally add
override_simulation_dir
to override the default_simulation_dir
for that specific run farm host.
Similarly, a specification can optionally add override_platform
to choose a different default_platform
for
that specific run farm host.
5.6. Build Farm Recipes (build-farm-recipes/*
)¶
Here is an example of this configuration file:
# Build-time build farm design configuration for the FireSim Simulation Manager
# See https://docs.fires.im/en/stable/Advanced-Usage/Manager/Manager-Configuration-Files.html for documentation of all of these params.
# all fields are required but can be overridden in the `*_runtime.yaml`
###########
# Schema:
###########
# # Class name of the build farm type.
# # This can be determined from `deploy/buildtools/buildfarm.py`).
# build_farm_type: <TYPE NAME>
# args:
# # Build farm arguments that are passed to the `BuildFarmHostDispatcher`
# # object. Determined by looking at `parse_args` function of class.
# <K/V pairs of args>
# Note: For large designs (ones that would fill a EC2 F1.2xlarge/Xilinx VU9P)
# Vivado uses in excess of 32 GiB. Keep this in mind when selecting a
# non-default instance type.
build_farm_type: AWSEC2
args:
# managerinit arg start
# tag to apply to build farm hosts
build_farm_tag: mainbuildfarm
# instance type to use per build
instance_type: z1d.2xlarge
# instance market to use per build (ondemand, spot)
build_instance_market: ondemand
# if using spot instances, determine the interrupt behavior (terminate, stop, hibernate)
spot_interruption_behavior: terminate
# if using spot instances, determine the max price
spot_max_price: ondemand
# default location of build directory on build host
default_build_dir: /home/centos/firesim-build
# managerinit arg end
5.6.1. build_farm_type
¶
This key/value specifies a build farm class to use for launching, managing, and terminating
build farm hosts used for building bitstreams.
By default, build farm classes can be found in deploy/buildtools/buildfarm.py. However, you can specify
your own custom build farm classes by adding your python file to the PYTHONPATH
.
For example, to use the AWSEC2
build farm class, you would write build_farm_type: AWSEC2
.
5.6.2. args
¶
This section specifies all arguments needed for the specific build_farm_type
used.
For a list of arguments needed for a build farm class, users should refer to
the _parse_args
function in the build farm class given by build_farm_type
.
5.6.3. aws_ec2.yaml
build farm recipe¶
This build farm recipe configures a FireSim build farm to use AWS EC2 instances enabled with Vivado.
Here is an example of this configuration file:
# Build-time build farm design configuration for the FireSim Simulation Manager
# See https://docs.fires.im/en/stable/Advanced-Usage/Manager/Manager-Configuration-Files.html for documentation of all of these params.
# all fields are required but can be overridden in the `*_runtime.yaml`
###########
# Schema:
###########
# # Class name of the build farm type.
# # This can be determined from `deploy/buildtools/buildfarm.py`).
# build_farm_type: <TYPE NAME>
# args:
# # Build farm arguments that are passed to the `BuildFarmHostDispatcher`
# # object. Determined by looking at `parse_args` function of class.
# <K/V pairs of args>
# Note: For large designs (ones that would fill a EC2 F1.2xlarge/Xilinx VU9P)
# Vivado uses in excess of 32 GiB. Keep this in mind when selecting a
# non-default instance type.
build_farm_type: AWSEC2
args:
# managerinit arg start
# tag to apply to build farm hosts
build_farm_tag: mainbuildfarm
# instance type to use per build
instance_type: z1d.2xlarge
# instance market to use per build (ondemand, spot)
build_instance_market: ondemand
# if using spot instances, determine the interrupt behavior (terminate, stop, hibernate)
spot_interruption_behavior: terminate
# if using spot instances, determine the max price
spot_max_price: ondemand
# default location of build directory on build host
default_build_dir: /home/centos/firesim-build
# managerinit arg end
5.6.3.1. build_farm_tag
¶
Use build_farm_tag
to differentiate between different Build Farms used across multiple FireSim repositories.
The instances launched by the buildbitstream
command will be tagged with
this value.
Mainly for CI use.
Per AWS restrictions, this tag can be no longer than 255 characters.
5.6.3.2. instance_type
¶
The AWS EC2 instance name to run a bitstream build on. Note that for large designs, Vivado uses an excess of 32 GiB so choose a non-default instance type wisely.
5.6.3.3. build_instance_market
¶
You can specify either spot
or ondemand
here, to use one of those
markets on AWS.
5.6.3.4. spot_interruption_behavior
¶
When run_instance_market: spot
, this value determines what happens to an instance
if it receives the interruption signal from AWS. You can specify either
hibernate
, stop
, or terminate
.
5.6.3.5. spot_max_price
¶
When build_instance_market: spot
, this value determines the max price you are
willing to pay per instance, in dollars. You can also set it to ondemand
to set your max to the on-demand price for the instance.
5.6.3.6. default_build_dir
¶
This is the path on the build farm host that bitstream builds will run out of.
5.6.4. externally_provisioned.yaml
build farm recipe¶
This build farm recipe allows users to provide an list of pre-setup unmanaged build farm hosts (by hostname or IP address) that they can run bitstream builds on. Note that this build farm type does not launch or terminate the build farm hosts. This functionality should be handled by the user. For example, users can use this build farm type to run bitstream builds locally.
Here is an example of this configuration file:
# Build-time build farm design configuration for the FireSim Simulation Manager
# See https://docs.fires.im/en/stable/Advanced-Usage/Manager/Manager-Configuration-Files.html for documentation of all of these params.
###########
# Schema:
###########
# # Class name of the build farm type.
# # This can be determined from `deploy/buildtools/buildfarm.py`).
# build_farm_type: <TYPE NAME>
# args:
# # Build farm arguments that are passed to the `BuildFarmHostDispatcher`
# # object. Determined by looking at `parse_args` function of class.
# <K/V pairs of args>
# Unmanaged list of build hosts. Assumed that they are pre-setup to run builds.
build_farm_type: ExternallyProvisioned
args:
# managerinit arg start
# REQUIRED: (replace this) default location of build directory on build host.
default_build_dir: null
# REQUIRED: List of IP addresses (or "localhost"). Each can have an OPTIONAL
# argument, called "override_build_dir", specifying to override the default
# build directory.
#
# Ex:
# build_farm_hosts:
# # use localhost and don't override the default build dir
# - localhost
# # use other IP address (don't override default build dir)
# - "111.111.1.111"
# # use other IP address (override default build dir for this build host)
# - "222.222.2.222":
# override_build_dir: /scratch/specific-build-host-build-dir
build_farm_hosts:
- localhost
# managerinit arg end
5.6.4.1. default_build_dir
¶
This is the default path on all the build farm hosts that bitstream builds will run out of.
5.6.4.2. build_farm_hosts
¶
This is a sequence of unique hostnames/IP addresses that should be used as build farm hosts.
Each build farm host (given by the unique hostname/IP address) can have an optional mapping that provides an
override_build_dir
that overrides the default_build_dir
given just for that build farm host.
5.7. Bit Builder Recipes (bit-builder-recipes/*
)¶
Here is an example of this configuration file:
# Build-time bitbuilder design configuration for the FireSim Simulation Manager
# See https://docs.fires.im/en/stable/Advanced-Usage/Manager/Manager-Configuration-Files.html for documentation of all of these params.
###########
# Schema:
###########
# # Class name of the bitbuilder type.
# # This can be determined from `deploy/buildtools/bitbuilder.py`).
# bitbuilder_type: <TYPE NAME>
# args:
# # Bitbuilder arguments that are passed to the `BitBuilder`
# # object. Determined by looking at `_parse_args` function of class.
# <K/V pairs of args>
bit_builder_type: F1BitBuilder
args:
# REQUIRED: name of s3 bucket
s3_bucket_name: firesim
# REQUIRED: append aws username and current region to s3_bucket_name?
append_userid_region: true
5.7.1. bit_builder_type
¶
This key/value specifies a bit builder class to use for building bitstreams.
By default, bit builder classes can be found in deploy/buildtools/bitbuilder.py. However, you can specify
your own custom bit builder classes by adding your python file to the PYTHONPATH
.
For example, to use the F1BitBuilder
build farm class, you would write bit_builder_type: F1BitBuilder
.
5.7.2. args
¶
This section specifies all arguments needed for the specific bit_builder_type
used.
For a list of arguments needed for a bit builder class, users should refer to
the _parse_args
function in the bit builder class given by bit_builder_type
.
5.7.3. f1.yaml
bit builder recipe¶
This bit builder recipe configures a build farm host to build an AWS EC2 F1 AGFI (FPGA bitstream).
Here is an example of this configuration file:
# Build-time bitbuilder design configuration for the FireSim Simulation Manager
# See https://docs.fires.im/en/stable/Advanced-Usage/Manager/Manager-Configuration-Files.html for documentation of all of these params.
###########
# Schema:
###########
# # Class name of the bitbuilder type.
# # This can be determined from `deploy/buildtools/bitbuilder.py`).
# bitbuilder_type: <TYPE NAME>
# args:
# # Bitbuilder arguments that are passed to the `BitBuilder`
# # object. Determined by looking at `_parse_args` function of class.
# <K/V pairs of args>
bit_builder_type: F1BitBuilder
args:
# REQUIRED: name of s3 bucket
s3_bucket_name: firesim
# REQUIRED: append aws username and current region to s3_bucket_name?
append_userid_region: true
5.7.3.1. s3_bucket_name
¶
This is used behind the scenes in the AGFI creation process. You will only ever need to access this bucket manually if there is a failure in AGFI creation in Amazon’s backend.
Naming rules: this must be all lowercase and you should stick to letters and numbers ([a-z0-9]).
The first time you try to run a build, the FireSim manager will try to create the bucket you name here. If the name is unavailable, it will complain and you will need to change this name. Once you choose a working name, you should never need to change it.
In general, firesim-yournamehere
is a good choice.
5.7.3.2. append_userid_region
¶
When enabled, this appends the current users AWS user ID and region to the s3_bucket_name
.
5.7.4. vitis.yaml
bit builder recipe¶
This bit builder recipe configures a build farm host to build an Vitis bitstream (FPGA bitstream called an xclbin
, packaged into a bitstream_tar
).
5.7.4.1. device
¶
This specifies a Vitis platform to compile against, for example: xilinx_u250_gen3x16_xdma_3_1_202020_1
when targeting a Vitis-enabled Alveo U250 FPGA.
Here is an example of this configuration file:
# Build-time bitbuilder design configuration for the FireSim Simulation Manager
# See https://docs.fires.im/en/stable/Advanced-Usage/Manager/Manager-Configuration-Files.html for documentation of all of these params.
###########
# Schema:
###########
# # Class name of the bitbuilder type.
# # This can be determined from `deploy/buildtools/bitbuilder.py`).
# bitbuilder_type: <TYPE NAME>
# args:
# # Bitbuilder arguments that are passed to the `BitBuilder`
# # object. Determined by looking at `_parse_args` function of class.
# <K/V pairs of args>
bit_builder_type: VitisBitBuilder
args:
# REQUIRED: vitis fpga platform
device: xilinx_u250_gen3x16_xdma_4_1_202210_1
5.7.5. xilinx_alveo_u250.yaml
bit builder recipe¶
This bit builder recipe configures a build farm host to build an Xilinx Alveo U250 bitstream, packaged into a bitstream_tar
.
5.7.6. xilinx_alveo_u280.yaml
bit builder recipe¶
This bit builder recipe configures a build farm host to build an Xilinx Alveo U280 bitstream, packaged into a bitstream_tar
.
5.7.7. xilinx_vcu118.yaml
bit builder recipe¶
This bit builder recipe configures a build farm host to build an Xilinx VCU118 bitstream, packaged into a bitstream_tar
.
5.7.8. rhsresearch_nitefury_ii.yaml
bit builder recipe¶
This bit builder recipe configures a build farm host to build an RHS Research Nitefury II bitstream, packaged into a bitstream_tar
.