1# Developing and running interop benchmarks in VMB
2
3## Preparing project
4
5- ensure your runtime core and ets frontend are on latest version
6- follow the regular steps to cmake the project:
7    - create a symbolic link from ets_frontend/ets2panda at runtime_core/static_core/tools/es2panda (note that e**t**s2panda points to es2panda) with `ln -s $(realpath ./ets_frontend/ets2panda) $(realpath ./runtime_core/static_core/tools/es2panda)`
8    - create `runtime_core/static_core/build` directory 
9    - run `cmake .. -GNinja -DCMAKE_BUILD_TYPE=debug -DCMAKE_EXPORTCOMPILECOMMANDS=ON -DPANDA_ETS_INTEROP_JS=1` from `runtime_core/static_core/build`
10- build binaries required by VMB:
11    `ninja es2panda ark ark_aot etsstdlib ets_interop_js_napi`
12
13- (optionally) build VMB via install-deps script to run it natively.
14```
15sudo ./static_core/scripts/install-deps-ubuntu --install=vmb
16```
17
18## Developing interop benchmarks
19
20### File structure and location
21
22All the interop benchmarks will be located in `runtime_core/static_core/plugins/ets/tests/benchmarks`. Example tests to be used as boilerplates can be found in `runtime_core/static_core/tests/vm-benchmarks/examples/benchmarks-interop-freestyle` 
23
24Within this folder, every benchmark suite is put into a subdirectory with a related name. This folder contains named separate benchmarks, or bench units - also in subdirectories, prefixed with "bu_". 
25
26Within the bench unit, VMB searches for files matching the pattern "bench_%benchUnitName%.(abc|zip|js|sts)", in this exact order.
27
28- if **abc | zip** files are found, VMB will assume these are precompiled binaries (or archives of such). Therefore, source compilation will be skipped, and the binaries will be executed as is.
29- **js** overrides default interop runner. Since that, it can be used BOTH for benchmarking pure javascript code AND custom interop logic.
30- If only **sts** files are found matching the pattern, they will compiled and executed using the default runner.  
31
32### Naming convention
33
34A benchmark suite should contain four bench units: pure JS, pure ArkTS, ArkTS invokes JS, JS invokes ArkTS. To distinguish those within a suite, bench units should be suffixed with `j2j`, `a2a`, `a2j`, or `j2a`, respectively. First letter represents the runtime invoking a certain feature and (optionally) handling the invocation outcome , while the latter indicates a runtime the feature is executed in.
35
36### Sample structure
37
38```
39*
40└── feature_name
41    ├── bu_feature_name_a2j
42    │   ├── bench_a2j.sts
43    │   └── test_import.js
44    ├── bu_feature_name_j2a
45    │   ├── bench_a2j.sts
46    │   └── bench_a2j.js
47    ├── bu_feature_name_j2j
48    │   └── bench_a2j.js
49    └── bu_feature_name_a2a
50        └── bench_j2a.sts
51
52```
53
54
55### Code structure
56
57### General tips:
58- VMB reads stdout for benchmark unit results. You'll have to console.log a certain pattern there to make results received and parsed properly: `Benchmark result: %YOUR_BENCHMARK_UNIT_NAME% %UNIT_TIMING%`
59- When using custom functions to time an iteration, please ensure timing calls are placed as close to the test calls as humanly possible.
60- `arktsconfig.json` is generated when a benchmark is ran. All JS files in a benchmark unit are automatically added to this JSON with path names as file name minus extension (e.g. `test_import.js` will be accessible as `test_import`)
61- use `Chrono.nanoNow` from `std/time` package in ARK, or `process.hrtime` in JS to provide better precision. 
62
63### *.sts:
64
65STS benchmark file should export a class with a name matching the overall bench unit name, but in pascal case (camelcase with a capital first letter). E.g., for a bench unit `call_async_function` a class `CallAsyncFunction` would be expected, while `SHA256` benchmark will need a class also called `SHA256`.
66
67####  The class **must** expose these public methods:
68
69- `setup(): void`
70
71This method is called once before the benchmark is ran and can be used either for warmup or preparing whatever is needed.
72
73
74- `test(): void`
75
76This method is called every benchmark iteration and basically IS the iteration.
77
78#### The class **may also** expose these public properties or methods
79
80- `runsLeft: number` (or an exposed setter for it)
81- `totalTime: number` (or an exposed getter)
82
83These properties indicate that VMB would expect iterations to be timed in this particular class (opposed to timing them externally). Since that, **please ensure every `test()` call is somehow timed, and iteration time is added to totalTime property**
84
85### *.js
86
87JS benchmark file is relatively free in structure, except for these two limitations:
88
89- VMB can not call a particular method inside a JS script by itself, so your code should be invoked on module load. E.g.:
90```
91function main() {
92    // RUN_SOME_CODE_HERE
93}
94main()
95``` 
96
97
98## Running benchmarks
99
100```
101export PANDA_BUILD=$(realpath %PATH TO YOUR static_core/build%)
102vmb run -p arkts_node_interop_host -v debug `pwd`/%PATH YO YOUR BENCHMARKS%
103```
104
105Notes:
106- if you didn't install vmb as a CLI app, you can use it via `./static_core/tests/vm-benchmarks/run-vmb.sh` with the same syntax
107