You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

165 lines
19 KiB

---
geometry: margin=2cm
---
# libsledge Binary Interfaces
libsledge is a \*.a static library (archive) that is statically linked with a \*.bc file generated by the aWsm compiler when compiling to a \*.so Linux shared library that can be loaded and executed by the sledgert runtime.
The static library internally implements the aWsm ABI in order to link to the \*.bc file generated by the aWsm compiler. [See the relevant documentation for this ABI here](../awsm/doc/abi.md).
libsledge defines a ABI between the sledgert runtime and a \*.so shared library containing an executable serverless function. This is distinct from the aWsm ABI.
# SLEdge \*.so serverless module
A SLEdge \*.so serverless module is generated by the latter portion of the aWsm/SLEdge toolchain.
The first portion of the toolchain is responsible for compiling a source program into a WebAssembly module. This is handled by standard compilers capable of emitting WebAssembly.
The second portion of the toolchain is the aWsm compiler, which generates a \*.bc file with a well defined ABI
The third portion of the toolchain is the LLVM compiler, which ingests a \*.bc file emitted by aWsm and the libsledge static library, and emits a SLEdge \*.so serverless module.
## Architecture
In order to reduce the overhead of calling sledgert functions, libsledge operates on global state of type `sledge_abi__wasm_module_instance` at `sledge_abi__current_wasm_module_instance`. This represents the global state of the wasm32 context executing on a sledgert worker core. The scheduler is responsible for populating these symbols before yielding execution to a serverless function.
The `sledge_abi__wasm_module_instance` structure includes the WebAssembly function table and the WebAssembly linear memory. This subset was selected because the author believes that use of function pointers and linear memory is frequent enough that LTO when compiling the \*.so file is beneficial.
All WebAssembly state
## WebAssembly Instruction Implementation
Here is a list of WebAssembly instructions that depend on symbols from libsledge, libc, or sledgert (via the SLEdge ABI).
### [Control Instructions](https://webassembly.github.io/spec/core/syntax/instructions.html#control-instructions)
The ABI includes the
Wasm traps (#301) * feat: Preliminary WASI with fib workload * refactor: Clarify initialize globals * chore: Update empty to WASI * chore: cleanup fib test * chore: cleanup build tooling * chore: cleanup test Makefiles and some nits * chore: Update LLVM and install WASI-SDK * chore: Update build tools and specs * docs: Update example module spec in README * refactor: Clean up HTTP handling * feat: Implement exit WASI call * style: apply clang-format * ci: rewrite compile sledge step * build: Remove LLVM install shims * build: Try manually adding libunwind * build: Try adding libunwind-dev * ci: break out aWsm compile step * fix: Correct test build error * fix: Correct error in WASI fd_write * chore: Increase gocr http buffer size * test: Correct image resize test * chore: Remove zombie wasmception functions * chore: Reduce dummy args to single arg * chore: Add debugging makefile fivebyeight * chore: Remove erronious PHONYs in tests Makefile * ci: Disable gocr tests * chore: Add wat Make rule to fibonacci test * chore: fix apt package name * chore: Enable clean of failed ck install * chore: use LLVM 12 * test: Disable gocr tests * chore: Enhance test makefile * chore: Add CFILES as sledgert dep * chore: Add NULL check for function table pointer * chore: Add missing header * chore: uncomment cleanup in imageresize test * refactor: Remove unused linear memory functions * build: Add bimodal debug makefile * chore: Add linear memory debug logs * refactor: Cleanup region initialization * build: Correct PHONY in runtime Makefile * chore: deb install script for outside of container * refactor: Remove zombie extern. * feat: WebAssembly traps * refactor: Use C18 features * chore: Remove git diff annotations * fix: tweaks to run all sample apps with WASI * test: convert shell script to Makefile * build: clean generated ck Makefile * chore: Use awsm branch with fixes * chore: Revert name changes * fix: Correct type issues * refactor: Reverse additional name change * refactor: Remove awsm compat shims * chore: Remove libc association * build: Better detect header file changes * refactor: current_wasm_module_instance_trap * test: reenable tests * chore: Delete copied script * build: Fix test workloads * fix: Implement HTTP 500 * fix: Protect against overflow on comparison * build: Replace test.sh with makefile * refactor: blow away tmp directory conflicts * refactor: centralize wasm in single submodule * feat: libsledge and sledge ABI * chore: move tests * refactor: tests * chore: update wasm_apps with new sample data * doc: Initial ABI README * feat: globals table * docs: Merge aWsm ABI docs * docs: libsledge ABI * build: rename apps to keep consistent * build: Disable wasm proposals * build: Update wasm apps and fix typo * test: test makefiles * test: Additional test makefiles * build: top-level build and install rules wo Docker * docs: Add wasm lld comment * build: top level makefile * chore: merge debug flags * fix: Correct out of bounds error * feat: indirection to awsm ABI * fix: Correct link hack with proper flag * fix: gps typo * chore: format nit * ci: update makefile rules * ci: check WASI_SDK_PATH * fix: Adjust paths * ci: fix make rule name * refactor: Attempt to use generic vec * refactor: Remove type-specific vec * fix: Resolve assorted TODOs * chore: fix clang format issue * ci: Invalidate app cache on libsledge changes * fix: Correct wasm trap check * fix: free wasm globals * docs: example of running top level tests via make * chore: option to log unsupported wasi * test: add preempt client generator for fib bimodal * refactor: Allocate wasm memory with 4096 align * fix: Handle build without runtime globals * refactor: bypass runtime call for first global * fix: Correct sandbox logging * test: fix incorrect paths in test.mk * refactor: cleanup basic traps logic * test: trap_divzero * fix: Correct inverted trap control flow * ci: Invalidate awsm cache * ci: divide by zero trap * chore: Assorted cleanup * ci: Install httpie * chore: Remove stdio calls in libsledge * chore: update submodules
3 years ago
| Instruction | aWsm ABI | libc Dependencies | SLEdge ABI |
| ------------- | ------------------------- | ------------------- | --------------------------------------------------------------------------------------------------------------------------------------- |
| call_indirect | `get_function_from_table` | `stderr`, `fprintf` | `sledge_abi__current_wasm_module_instance.table`, `sledge_abi__wasm_trap_raise`, `WASM_TRAP_INVALID_INDEX`, `WASM_TRAP_MISMATCHED_TYPE` |
### [Variable Instructions](https://webassembly.github.io/spec/core/syntax/instructions.html#variable-instructions)
| Instruction | aWsm ABI | libc Dependencies | SLEdge ABI |
| ----------- | ---------------------------------- | ----------------- | ---------------------------------------------------------------------- |
| global.get | `get_global_i32`, `get_global_i64` | None | `sledge_abi__wasm_globals_get_i32`, `sledge_abi__wasm_globals_get_i64` |
| global.set | `set_global_i32`, `set_global_i64` | None | `sledge_abi__wasm_globals_set_i32`, `sledge_abi__wasm_globals_set_i64` |
### [Numeric Instructions](https://webassembly.github.io/spec/core/syntax/instructions.html#numeric-instructions)
| Instruction | aWsm ABI | libc Dependencies | SLEdge ABI |
| --------------- | ---------------------------------------- | --------------------------------------------- | ----------------------------------------------------------------------- |
| i32.div_s | `i32_div` ("fast unsafe" disabled) | `stderr`, `fprintf` | `sledge_abi__wasm_trap_raise`, `WASM_TRAP_ILLEGAL_ARITHMETIC_OPERATION` |
| i32.div_u | `u32_div` ("fast unsafe" disabled) | `stderr`, `fprintf` | `sledge_abi__wasm_trap_raise`, `WASM_TRAP_ILLEGAL_ARITHMETIC_OPERATION` |
| i32.rem_s | `i32_rem` ("fast unsafe" disabled) | `stderr`, `fprintf` | `sledge_abi__wasm_trap_raise`, `WASM_TRAP_ILLEGAL_ARITHMETIC_OPERATION` |
| i32.rem_u | `u32_rem` ("fast unsafe" disabled) | `stderr`, `fprintf` | `sledge_abi__wasm_trap_raise`, `WASM_TRAP_ILLEGAL_ARITHMETIC_OPERATION` |
| i32.rotl | `rotl_u32` | None | None |
| i32.rotr | `rotr_u32` | None | None |
| i32.trunc_f32_s | `i32_trunc_f32` ("fast unsafe" disabled) | `stderr`, `fprintf` | `sledge_abi__wasm_trap_raise`, `WASM_TRAP_ILLEGAL_ARITHMETIC_OPERATION` |
| i32.trunc_f32_u | `u32_trunc_f32` ("fast unsafe" disabled) | `stderr`, `fprintf` | `sledge_abi__wasm_trap_raise`, `WASM_TRAP_ILLEGAL_ARITHMETIC_OPERATION` |
| i32.trunc_f64_s | `i32_trunc_f64` ("fast unsafe" disabled) | `INT32_MIN`, `INT32_MAX`, `stderr`, `fprintf` | `sledge_abi__wasm_trap_raise`, `WASM_TRAP_ILLEGAL_ARITHMETIC_OPERATION` |
| i32.trunc_f64_u | `u32_trunc_f64` ("fast unsafe" disabled) | `UINT32_MAX`, `stderr`, `fprintf` | `sledge_abi__wasm_trap_raise`, `WASM_TRAP_ILLEGAL_ARITHMETIC_OPERATION` |
| i64.div_s | `i64_div` ("fast unsafe" disabled) | `stderr`, `fprintf` | `sledge_abi__wasm_trap_raise`, `WASM_TRAP_ILLEGAL_ARITHMETIC_OPERATION` |
| i64.div_u | `u64_div` ("fast unsafe" disabled) | `stderr`, `fprintf` | `sledge_abi__wasm_trap_raise`, `WASM_TRAP_ILLEGAL_ARITHMETIC_OPERATION` |
| i64.rem_s | `i64_rem` ("fast unsafe" disabled) | `INT32_MIN`, `stderr`, `fprintf` | `sledge_abi__wasm_trap_raise`, `WASM_TRAP_ILLEGAL_ARITHMETIC_OPERATION` |
| i64.rem_u | `u64_rem` ("fast unsafe" disabled) | `stderr`, `fprintf` | `sledge_abi__wasm_trap_raise`, `WASM_TRAP_ILLEGAL_ARITHMETIC_OPERATION` |
| i64.rotl | `rotl_u64` | **NOT SUPPORTED** | **NOT SUPPORTED** |
| i64.rotr | `rotr_u64` | **NOT SUPPORTED** | **NOT SUPPORTED** |
| i64.trunc_f32_s | `i64_trunc_f32` | `INT64_MIN`, `INT64_MAX`, `stderr`, `fprintf` | `sledge_abi__wasm_trap_raise`, `WASM_TRAP_ILLEGAL_ARITHMETIC_OPERATION` |
| i64.trunc_f32_u | `u64_trunc_f32` | `UINT64_MAX`, `stderr`, `fprintf` | `sledge_abi__wasm_trap_raise`, `WASM_TRAP_ILLEGAL_ARITHMETIC_OPERATION` |
| i64.trunc_f64_s | `i64_trunc_f64` | `INT64_MIN`, `INT64_MAX`, `stderr`, `fprintf` | `sledge_abi__wasm_trap_raise`, `WASM_TRAP_ILLEGAL_ARITHMETIC_OPERATION` |
| i64.trunc_f64_u | `u64_trunc_f64` | `UINT64_MAX`, `stderr`, `fprintf` | `sledge_abi__wasm_trap_raise`, `WASM_TRAP_ILLEGAL_ARITHMETIC_OPERATION` |
| f32.ceil | `f32_ceil` | `ceilf` | `ceilf` |
| f32.copysign | `f32_copysign` | `copysignf` | `copysignf` |
| f32.floor | `f32_floor` | `floorf` | `floorf` |
| f32.max | `f32_max` | None | None |
| f32.min | `f32_min` | None | None |
| f32.nearest | `f32_nearest` | `nearbyintf` | `nearbyintf` |
| f32.trunc | `f32_trunc_f32` | `truncf` | `truncf` |
| f64.ceil | `f64_ceil` | `ceil` | `ceil` |
| f64.copysign | `f64_copysign` | `copysign` | `copysign` |
| f64.floor | `f64_floor` | `floor` | `floor` |
| f64.max | `f64_max` | None | None |
| f64.min | `f64_min` | None | None |
| f64.nearest | `f64_nearest` | `nearbyint` | `nearbyint` |
| f64.trunc | `f64_trunc_f64` | `trunc` | `trunc` |
### [Memory Instructions](https://webassembly.github.io/spec/core/syntax/instructions.html#memory-instructions)
| instruction | aWsm ABI | libc Dependencies | SLEdge ABI |
| ------------ | ------------------------- | ------------------- | ------------------------------------------------------------------------------------------------------------------------- |
| i32.load | `get_i32` | `fprintf`, `stderr` | `sledge_abi__current_wasm_module_instance.memory`, `sledge_abi__wasm_trap_raise`, `WASM_TRAP_OUT_OF_BOUNDS_LINEAR_MEMORY` |
| i32.load8_s | `get_i8` | `fprintf`, `stderr` | `sledge_abi__current_wasm_module_instance.memory`, `sledge_abi__wasm_trap_raise`, `WASM_TRAP_OUT_OF_BOUNDS_LINEAR_MEMORY` |
| i32.load8_u | `get_i8` | `fprintf`, `stderr` | `sledge_abi__current_wasm_module_instance.memory`, `sledge_abi__wasm_trap_raise`, `WASM_TRAP_OUT_OF_BOUNDS_LINEAR_MEMORY` |
| i32.load16_s | `get_i16` | `fprintf`, `stderr` | `sledge_abi__current_wasm_module_instance.memory`, `sledge_abi__wasm_trap_raise`, `WASM_TRAP_OUT_OF_BOUNDS_LINEAR_MEMORY` |
| i32.load16_u | `get_i16` | `fprintf`, `stderr` | `sledge_abi__current_wasm_module_instance.memory`, `sledge_abi__wasm_trap_raise`, `WASM_TRAP_OUT_OF_BOUNDS_LINEAR_MEMORY` |
| i32.store | `set_i32` | `fprintf`, `stderr` | `sledge_abi__current_wasm_module_instance.memory`, `sledge_abi__wasm_trap_raise`, `WASM_TRAP_OUT_OF_BOUNDS_LINEAR_MEMORY` |
| i32.store8 | `set_i8` | `fprintf`, `stderr` | `sledge_abi__current_wasm_module_instance.memory`, `sledge_abi__wasm_trap_raise`, `WASM_TRAP_OUT_OF_BOUNDS_LINEAR_MEMORY` |
| i32.store16 | `set_i16` | `fprintf`, `stderr` | `sledge_abi__current_wasm_module_instance.memory`, `sledge_abi__wasm_trap_raise`, `WASM_TRAP_OUT_OF_BOUNDS_LINEAR_MEMORY` |
| i64.load | `get_i64` | `fprintf`, `stderr` | `sledge_abi__current_wasm_module_instance.memory`, `sledge_abi__wasm_trap_raise`, `WASM_TRAP_OUT_OF_BOUNDS_LINEAR_MEMORY` |
| i64.load8_s | `get_i8` | `fprintf`, `stderr` | `sledge_abi__current_wasm_module_instance.memory`, `sledge_abi__wasm_trap_raise`, `WASM_TRAP_OUT_OF_BOUNDS_LINEAR_MEMORY` |
| i64.load8_u | `get_i8` | `fprintf`, `stderr` | `sledge_abi__current_wasm_module_instance.memory`, `sledge_abi__wasm_trap_raise`, `WASM_TRAP_OUT_OF_BOUNDS_LINEAR_MEMORY` |
| i64.load16_s | `get_i16` | `fprintf`, `stderr` | `sledge_abi__current_wasm_module_instance.memory`, `sledge_abi__wasm_trap_raise`, `WASM_TRAP_OUT_OF_BOUNDS_LINEAR_MEMORY` |
| i64.load16_u | `get_i16` | `fprintf`, `stderr` | `sledge_abi__current_wasm_module_instance.memory`, `sledge_abi__wasm_trap_raise`, `WASM_TRAP_OUT_OF_BOUNDS_LINEAR_MEMORY` |
| i64.load32_s | `get_i32` | `fprintf`, `stderr` | `sledge_abi__current_wasm_module_instance.memory`, `sledge_abi__wasm_trap_raise`, `WASM_TRAP_OUT_OF_BOUNDS_LINEAR_MEMORY` |
| i64.load32_u | `get_i32` | `fprintf`, `stderr` | `sledge_abi__current_wasm_module_instance.memory`, `sledge_abi__wasm_trap_raise`, `WASM_TRAP_OUT_OF_BOUNDS_LINEAR_MEMORY` |
| i64.store | `set_i64` | `fprintf`, `stderr` | `sledge_abi__current_wasm_module_instance.memory`, `sledge_abi__wasm_trap_raise`, `WASM_TRAP_OUT_OF_BOUNDS_LINEAR_MEMORY` |
| i64.store8 | `set_i8` | `fprintf`, `stderr` | `sledge_abi__current_wasm_module_instance.memory`, `sledge_abi__wasm_trap_raise`, `WASM_TRAP_OUT_OF_BOUNDS_LINEAR_MEMORY` |
| i64.store16 | `set_i16` | `fprintf`, `stderr` | `sledge_abi__current_wasm_module_instance.memory`, `sledge_abi__wasm_trap_raise`, `WASM_TRAP_OUT_OF_BOUNDS_LINEAR_MEMORY` |
| i64.store32 | `set_i32` | `fprintf`, `stderr` | `sledge_abi__current_wasm_module_instance.memory`, `sledge_abi__wasm_trap_raise`, `WASM_TRAP_OUT_OF_BOUNDS_LINEAR_MEMORY` |
| f32.load | `get_f32` | `fprintf`, `stderr` | `sledge_abi__current_wasm_module_instance.memory`, `sledge_abi__wasm_trap_raise`, `WASM_TRAP_OUT_OF_BOUNDS_LINEAR_MEMORY` |
| f32.store | `set_f32` | `fprintf`, `stderr` | `sledge_abi__current_wasm_module_instance.memory`, `sledge_abi__wasm_trap_raise`, `WASM_TRAP_OUT_OF_BOUNDS_LINEAR_MEMORY` |
| f64.load | `get_f64` | `fprintf`, `stderr` | `sledge_abi__current_wasm_module_instance.memory`, `sledge_abi__wasm_trap_raise`, `WASM_TRAP_OUT_OF_BOUNDS_LINEAR_MEMORY` |
| f64.store | `set_f64` | `fprintf`, `stderr` | `sledge_abi__current_wasm_module_instance.memory`, `sledge_abi__wasm_trap_raise`, `WASM_TRAP_OUT_OF_BOUNDS_LINEAR_MEMORY` |
| memory.grow | `instruction_memory_grow` | | `sledge_abi__current_wasm_module_instance.memory`, `sledge_abi__wasm_memory_expand` |
| memory.size | `instruction_memory_size` | | `sledge_abi__current_wasm_module_instance.memory` |
| None | `initialize_region` | | `sledge_abi__current_wasm_module_instance.memory`, `sledge_abi__wasm_memory_initialize_region` |
Discussion:
- Should `instruction_memory_grow` be moved into sledgert? This would simplify the handling of the "cache" and generating a memory profile?
- Rename `sledge_abi__wasm_globals_*` to `sledge_abi__wasm_global_*`
- Implement Unsupported Numeric Instructions
- Should the wasm global table be accessed directly instead of via a runtime function?
- Should the Function Table be handled by the \*.so file or sledgert? Are function pointers really called that frequently?
# SLEdge \*.so Module Loading / Initialization
The `sledgert` runtime is invoked with an argument containing the path to a JSON file defining serverless functions. The JSON format is a top level array containing 0..N JSON objects with the following keys:
"name" - A Human friendly name identifying a serverless function. Is required.
"path" - A path to a \*.so module containing the program to be executed
"port" - The port which the serverless function is registered
"relative-deadline-us"
"expected-execution-us"
"admissions-percentile"
"http-req-size"
"http-resp-size"
"http-resp-content-type"
The path to the JSON file is passed to `module_alloc_from_json`, which uses the Jasmine library to parse the JSON, performs validation, and passes the resulting specification to `module_alloc` for each module definition found. `module_alloc` allocated heap memory for a `struct module` and then calls `module_init`. `module_init` calls `sledge_abi_symbols_init`, which calls `dlopen` on the _.so file at the path specified in the JSON and then calls `dlsym` to resolve symbols within the _.so module.
`module.abi.initialize_globals` -> `SLEDGE_ABI__INITIALIZE_GLOBALS` -> `populate_globals`
`module.abi.initialize_memory`-> `SLEDGE_ABI__INITIALIZE_MEMORY` -> `populate_memory`
`module.abi.initialize_table` -> `SLEDGE_ABI__INITIALIZE_TABLE` -> `populate_table`
`module.abi.entrypoint` -> `SLEDGE_ABI__ENTRYPOINT` -> `wasmf__start`
`module.abi.starting_pages` -> `SLEDGE_ABI__STARTING_PAGES` -> `starting_pages`
`module.abi.max_pages` -> `SLEDGE_ABI__MAX_PAGES` -> `max_pages`
`module.abi.globals_len` -> `SLEDGE_ABI__GLOBALS_LEN` -> `globals_len`
`module init` then calls `module.abi.initialize_table`, which populates the indirect function table with the actual functions. This is performed once during module initialization because this table does not actually vary between instances of a module.
# SLEdge \*.so Module Instantiation
When `sledgert` receives a request at the registered port specified in the JSON, it performs assorted allocation and initialization steps. The scheduler sets the expected ABI symbols and yields to `current_sandbox_start`, which immediately calls `current_sandbox_init`. This function initializes the associated runtime state and
1. calls `module.abi.initialize_globals` for the current sandbox if not NULL. This is optional because the module might not have been built with the `--runtime-globals`, in which case runtime globals are not used at all. If not NULL, the globals are set in the table.
2. calls `module.abi.initialize_memory`, which copies regions into the linear memory
`current_sandbox_init` calls `wasi_context_init` to initialize the WASI context within the runtime.
`current_sandbox_init` returns to `current_sandbox_start`, which sets up wasm traps using `setjmp` and then calls `module.abi.entrypoint`
# Questions:
- Should `sledge_abi__current_wasm_module_instance` be turned into a macro defined int the ABI header?