1# VM Benchmarks 2 3VMB is a tool for running benchmarks for variety of Virtual Machines, 4platforms and languages. 5 6## Quick start 7 8The only requirement is python3 (3.7+), no additional modules needed. 9Virtual Machine to test should be installed on host and/or device. 10See [Notes on setup for platforms](#platforms). 11 12#### Option 1: Using wrapper script 13 14This option is to try VMB without any installation involved. 15 16```shell 17# Run all js and ts examples on Node (needs node and tsc): 18./run-vmb.sh all -p node_host `pwd`/examples/benchmarks 19 20# Run all only js examples on Node (needs node installed): 21./run-vmb.sh all -l js -p node_host `pwd`/examples/benchmarks 22 23# See all options explained: 24./run-vmb.sh help 25 26# List all available plugins: 27./run-vmb.sh list 28``` 29 30#### Option2: Build and install vmb module (preferred) 31 32Wrapper script `run-vmb.sh` has all the module functionality, 33but it requires absolute paths in cmdline. 34Python package, built and installed, provides more flexibility 35in dealing with various benchmark sources and repos. 36Once installed it exposes single `vmb` command for generating, 37running and reporting tests. 38 39```sh 40# This command will build and install VMB python module on your machine. 41# No root permissions required. 42# After that `vmb` cli will appear in your $PATH 43 44make vmb 45 46# Run all js and ts tests in current dir 47vmb all -p node_host 48 49# Run only js tests with tag 'sanity' from 'examples' 50vmb all -p node_host -l js -T sanity ./examples 51``` 52 53#### Usage Example: Compare 2 runs 54 55```shell 56export PANDA_BUILD=~/arkcompiler/runtime_core/static_core/build 57# Run sts and ts tests on ArkTS in 58# 1) interpretation mode 59vmb all -p arkts_host --aot-skip-libs \ 60 --mode=int --report-json=int.json ./examples/benchmarks/ 61# 2) in JIT mode 62vmb all -p arkts_host --aot-skip-libs \ 63 --mode=jit --report-json=jit.json ./examples/benchmarks/ 64# Compare results 65vmb report --compare int.json jit.json 66 67 Comparison: arkts_host-int vs arkts_host-jit 68 ======================================== 69 Time: 1.99e+05->3.80e+04(better -80.9%); Size: 9.17e+04->9.17e+04(same); RSS: 4.94e+04->5.34e+04(worse +8.1%) 70 ============================================================================================================= 71``` 72 73## Commands: 74`vmb` cli supports following commands: 75- `gen` - generate benchmark tests from [doclets](#doclet-format) 76- `run` - run tests generated by `gen` or [non doclet tests](#non-doclet-tests) 77- `all` = `gen` + `run` 78- `report` - works with json report (display, compare) 79- `list` - show info supported langs and platforms 80- `help` - print options for all the commands. (Also `vmb <cmd> --help`) 81 82## Selecting platform 83Required `-p` (`--platform`) option to `run` or `all` command should be the name 84of plugin from `<vmb-package-root>/plugins/platforms` 85or from `<extra-plugins-dir>/platforms` if specified. 86F.e. `ark_js_vm_host` - for ArkHz on host machine, 87or `arkts_ohos` - for ArkTS on OHOS device 88 89## Selecting language and source files 90`gen` command requires `-l` (`--langs`) option. 91F.e. `vmb gen -l sts,swift,ts,js ./examples/benchmarks` 92will generate benches for all 4 languages in examples. 93 94Then provided to `all` command `--langs` will override langs, supported by platform. 95F.e. `vmb all -l js -p v_8_host ./tests` will skip `ts` (typescript) 96 97Source files (doclets) could be overridden 98by `--src-langs` option to `all` or `run` command. 99 100Each platform defines languages it can deal with, 101and each language defines its source file extension. 102Defaults are: 103 104| platform | langs | sources | 105|---------------|------------|-----------------| 106| `arkts_*` | `sts` | `*.sts`, `*.ts` | 107| `ark_js_vm_*` | `ts` | `*.ts` | 108| `swift_*` | `swift` | `*.swift` | 109| `v_8_*` | `ts`, `js` | `*.ts`, `*.js` | 110| `node_*` | `ts`, `js` | `*.ts`, `*.js` | 111 112## Selecting and filtering tests: 113- Any positional argument to `all` or `gen` command would be treated 114as path to doclets: `vmb all -p x ./test1/test1.js ./more-tests/ ./even-more` 115- Any positional argument to `run` command would be treated 116as path to generated tests: `vmb run -p x ./generated` 117- Files with `.lst` extension would be treated as lists of paths, relative to `CWD` 118- `--tags=sanity,my` (`-T`) will generate and run only tests tagged with `sanity` OR `my` 119- `--tests=my_test` (`-t`) will run only tests containing `my_test` in name 120- To exclude some tests from run point `--exclude-list` to the file with test names to exclude 121- To exclude tests with tags use `-ST` (`--skip-tags`) option. F.e. `--skip-tags=negative,flaky` 122 123## Benchmark's measurement options: 124* `-wi` (`--warmup-iters`) controls the number of warmup iterations, 125 default is 2. 126* `-mi` (`--measure-iters`) controls the number of measurement iterations, 127 default is 3. 128* `-it` (`--iter-time`) controls the duration of iterations in seconds, 129 default is 1. 130* `-wt` (`--warmup-time`) controls the duration of warmup iterations in seconds, 131 default is 1. 132* `-gc` (`--sys-gc-pause`) Non-negative value forces GC (twice) 133 and <value> milliseconds sleep before each iteration 134 (GC finish can't be guaranteed on all VM's), default is -1 (no forced GC). 135* `-fi` (`--fast-iters`) Number of 'fast' iterations 136 (no warmup, no tuning cycles). 137 Benchmark will run this number of iterations, regardless of time elapsed. 138 139## Custom options 140To provide additional option to compiler or virtual machine 141`--<tool>-custom-option` could be used. F.e. add cpu profiling for `node`: 142`vmb all -p node_host --node-custom-option='--cpu-prof' -v debug ./examples/benchmarks/ts/VoidBench.ts` 143 144 145## Reports: 146* `--report-json=path.json` to save results in json format, 147 this is most detailed report. 148 `--report-json-compact` disables prettifying of json. 149* `--report-csv=path.csv` to save results in csv format. Only basic info included. 150 151## Controlling log: 152There are several log levels which could be set via `--log-level` option. 153- `fatal`: only critical errors will be printed 154- `pass`: print single line for each test after it finishes 155- `error` 156- `warn` 157- `info`: default level 158- `debug` 159- `trace`: most verbose 160 161Each level prints in its own color. 162`--no-color` disables color and adds `[%LEVEL]` prefix to each message. 163 164## Extra (custom) plugins: 165 166Providing option `--extra-plugins` you can add any 167language, tool, platform or hook. 168This parameter should point to the directory containing any combination of 169`hooks`, `langs`, `platforms`, `tools` sub-dirs. 170As the simplest case you can copy any of the existing plugins 171and safely experiment modifying it. 172`extra-plugins` has higher 'priority', so any tools, platforms and langs 173could be 'overridden' by custom ones. 174 175```sh 176# Example: 177# 2 demo plugins: Tool=dummy and Platform=dummy_host 178mkdir bu_Fake 179touch bu_Fake/bench_Fake.txt 180vmb run -p dummy_host \ 181 --extra-plugins=./examples/plugins \ 182 ./bu_Fake/bench_Fake.txt 183``` 184 185## Doclet format: 186This is a mean for benchmark declaration in source file. 187In a single root class it is possible to define any number of benchmarks, 188setup method, benchmarks parameters, run options and some additional meta-info. 189See examples in `examples/benchamrks` directory. 190 191Using this test format involves separate generation stage (`gen` command). 192Resulting benchmark programs will be generated using templates 193in `plugins/templates` into `generated` directory 194(which could be overriden by `--outdir` option) 195 196Supported doclets are: 197* `@State` on a root class which contains benchmarks tests as its methods. 198* `@Setup` on an initialization method of a state. 199* `@Benchmark` on a method that is measured. 200 It should not accept any parameters and 201 (preferably) it may return a value which is consumed. 202* `@Param p1 [, p2...]` on a state field. 203 Attribute define values to create several benchmarks using same code, 204 and all combinations of params. 205 Value can be an int, a string or a comma separated list of ints or strings. 206* `@Tags t1 [, t2...]` on a root class or method. List of benchmark tags. 207* `@Bugs b1 [, b2...]` on a root class or method. List of associated issues. 208 209JS specific doclets (JSDoc-like) are: 210* `@arg {<State name>} <name>` on a benchmark method. 211 Links a proper state class to each parameter. 212* `@returns {Bool|Char|Int|Float|Obj|Objs}` optionally on a benchmark method. 213 Helps to build automatic `Consumer` usage. 214 215## Non-doclet tests 216 217It is possible to write 'free-style' tests which can be ran as is. 218VMB will search bench units in directories with `bu_` prefix and files with 219`bench_` prefix. 220Test result should be reported by program 221as `Benchmark result: <TestName> <time in ns>`. 222 223## Platforms 224 225### ArkTS (arkts_*) 226On host: env var `PANDA_BUILD` should be set to point to `build` directory 227f.e. `~/arkcompiler/runtime_core/static_core/build` 228 229On device: `ark` and `ark_aot` binaries with required libraries should be pushed to device. 230Default place is `/data/local/tmp/vmb` and could be configured via `--device-dir` 231 232By default `etsstdlib` will be compiled into native code, 233to avoid this use `--aot-skip-libs` (`-A`) option 234 235### Ark JS VM (ark_js_vm_*) 236On host: env var `OHOS_SDK` sould be set to point to OHOS SDK source directory 237inside which `out/*.release` should contain built binaries 238 239On device: `ark_js_vm` and `ark_aot_compiler` should be pushed to device. 240Default place is `/data/local/tmp/vmb` and could be configured via `--device-dir` 241 242### V_8 243For device platforms: required binaries should be pushed to device. 244Default place is `/data/local/tmp/vmb/v_8`. 245`/data/local/tmp/vmb` part could be configured via `--device-dir` 246 247## Platform Features 248 249| platform | int-mode | aot-mode | jit-mode | gc-stats | jit-stats | aot-stats | imports | 250|-------------------|:--------:|:--------:|:--------:|:--------:|:---------:|:---------:|:-------:| 251| ark_js_vm_host | n/a | V | V | n/a | n/a | n/a | V | 252| ark_js_vm_ohos | n/a | V | V | n/a | n/a | n/a | V | 253| arkts_device | V | V | V | V | V | V | V | 254| arkts_host | V | V | V | V | V | V | V | 255| arkts_ohos | V | V | V | V | V | V | V | 256| node_host | n/a | n/a | V | n/a | n/a | n/a | V | 257| swift_device | n/a | n/a | n/a | n/a | n/a | n/a | X | 258| swift_host | n/a | n/a | n/a | n/a | n/a | n/a | X | 259| v_8_device | V | n/a | V | n/a | n/a | n/a | V | 260| v_8_host | V | n/a | V | n/a | n/a | n/a | V | 261| v_8_ohos | V | n/a | V | n/a | n/a | n/a | V | 262 263 264## Interoperability tests: 265 266#### Freestyle 267Please refer to [this manual](./interop.readme.md) for freestyle benchmark development and running guide. 268 269#### Doclet 270Allows use corpus of existing VMB tests in interop mode. 271Benchmarks are generated from sts [doclets](#doclet-format). 272 273```shell 274export PANDA_BUILD=$HOME/ark-built-with-interop/runtime_core/static_core/build 275vmb all -p arkts_node_interop_host -v debug ./examples/benchmarks/sts 276``` 277 278## Self tests and linters 279```sh 280make tox 281```