diff --git a/docs/advanced/evaluation_functions/alternate_languages.md b/docs/advanced/evaluation_functions/alternate_languages.md index 60641aa1a..88beb493d 100644 --- a/docs/advanced/evaluation_functions/alternate_languages.md +++ b/docs/advanced/evaluation_functions/alternate_languages.md @@ -1,37 +1,77 @@ -# Alternate Evaluation Function Languages ---- - -## Lambda-Compatible Images -### Extending a pre-built Lambda image -- Available for: Node.js, Python, Java, .NET, Go, Ruby -- [Docs](https://docs.aws.amazon.com/lambda/latest/dg/runtimes-images.html#runtimes-images-lp) -- [Repo](https://github.com/aws/aws-lambda-base-images) -- These base images are regularly updated, and the most widely used (more docs) -- They also come with pre-packaged runtime interface clients - a HTTP interface for runtimes to receive invocation events and respond - - Good for local development - -### Creating custom base images -- Using the [lambda/provided](https://gallery.ecr.aws/lambda/provided) image - - This "contains all the required components to run functions packaged as container images on Lambda" -- Building a custom runtime from scratch - - [Custom AWS Lambda runtimes](https://docs.aws.amazon.com/lambda/latest/dg/runtimes-custom.html#runtimes-custom-build) - - [Runtimes walkthrough tutorial](https://docs.aws.amazon.com/lambda/latest/dg/runtimes-walkthrough.html) -- Emulate execution locally? -> Lambda provides a runtime interface emulator (RIE) for you to test your function locally. The AWS base images for Lambda and base images for custom runtimes include the RIE. For other base images, you can download the [Runtime interface emulator](https://github.com/aws/aws-lambda-runtime-interface-emulator) from the AWS GitHub repository. - -### Misc Notes/Sources -- [The Lambda Execution Environment](https://docs.aws.amazon.com/lambda/latest/dg/lambda-runtime-environment.html) -- [Create Images from Alternative base images](https://docs.aws.amazon.com/lambda/latest/dg/images-create.html#images-create-from-alt) - -## Development Philosophy -Ultimately we want to call a function made by a user in any language. Two ways to do this: - -- We write and provide runtime in all the different languages. This means that all the logic happens in that language. We write the code that actually receives the requests from lambda function events. In this case, the user function can be imported from those handlers. - - Writing handlers in each of those languages requires time and extensive knowledge (in order to write robust code) - - Handler code needs to: - - Have clean and reliable error catching - -- We write a global runtime, which makes a call to their function via a sub-process. We call their script, which must recieve the payload as a commandline argument. - - User has to write more code - - For allowing cmdline arguments, and parsing of inputs - - Might be slower than in other languages. Since another script has to be executed. \ No newline at end of file +# Evaluation Functions in Other Languages + +[Shimmy](https://github.com/lambda-feedback/shimmy) — the [base layer](specification.md#base-layer) +in front of every evaluation function — is language-agnostic. It handles the HTTP API, request +validation and the feedback `cases` loop, then runs *your* function as a child process over one +of two interfaces. Writing a function in another language means providing that child process. + +## Base images + +All base images bundle Shimmy and are published under +[`ghcr.io/lambda-feedback/evaluation-function-base`](https://github.com/lambda-feedback/evaluation-function-base): + +| Image | For | +| --- | --- | +| `evaluation-function-base/python` | Python functions (uses [`lf_toolkit`](module.md)) | +| `evaluation-function-base/wolfram` | Wolfram Language / `wolframscript` functions | +| `evaluation-function-base/lean` | Lean functions (compiled binary) | +| `evaluation-function-base/scratch` | Any other language — a minimal Debian image with just Shimmy | + +Your `Dockerfile` does `FROM` one of these, installs your toolchain and code, and sets the +environment variables below. + +## Worker interfaces + +Shimmy chooses the interface from the `FUNCTION_INTERFACE` environment variable. + +### RPC (default) + +The worker is a long-lived process that speaks [JSON-RPC 2.0](https://www.jsonrpc.org/specification), +one method per command (`eval`, `preview`, `healthcheck`). Transport is set by +`FUNCTION_RPC_TRANSPORT`: + +- `stdio` (default) — messages over the process's stdin/stdout, framed with `Content-Length` headers; +- `ipc` — a Unix domain socket. + +Python's [`lf_toolkit`](module.md) implements this interface, so Python functions just call +`create_server()` / `run()` in `evaluation_function/main.py` and never deal with the wire format. +The Wolfram base image bundles [`toolkit-wolfram`](https://github.com/lambda-feedback/toolkit-wolfram), +which handles the transport wiring for `wolframscript` functions in the same way. + +### File + +Shimmy starts a **fresh process per request**, appending two paths as the final arguments — an +input file and an output file. The worker reads the request JSON, writes the response JSON and +exits. This suits languages without a convenient long-running-server story, and large payloads +(e.g. base64 images). + +The request file is *wrapped*: + +```json +{ + "command": "eval", + "params": { "response": "...", "answer": "...", "params": {} } +} +``` + +The worker writes the same `{"command": ..., "result": {...}}` / `{"error": {...}}` shape the +[Legacy API](specification.md#legacy-api) returns. + +## Setting the worker command + +The base layer reads these from the `Dockerfile`: + +```dockerfile +ENV FUNCTION_COMMAND="wolframscript" +ENV FUNCTION_ARGS="-f,evaluation_function.wl" # comma-separated +ENV FUNCTION_INTERFACE="file" +``` + +## Boilerplates + +- [`evaluation-function-boilerplate-python`](https://github.com/lambda-feedback/evaluation-function-boilerplate-python) — RPC interface via `lf_toolkit` +- [`evaluation-function-boilerplate-wolfram`](https://github.com/lambda-feedback/evaluation-function-boilerplate-wolfram) — file interface, `wolframscript -f evaluation_function.wl request.json response.json` +- [`evaluation-function-boilerplate-lean`](https://github.com/lambda-feedback/evaluation-function-boilerplate-lean) — file interface, compiled `.lake/build/bin/evaluation request.json response.json` + +Each boilerplate's `README.md` has the full build, run and local-test instructions for that +language. diff --git a/docs/advanced/evaluation_functions/feedback.md b/docs/advanced/evaluation_functions/feedback.md index 996f5c260..e2dd7b85b 100644 --- a/docs/advanced/evaluation_functions/feedback.md +++ b/docs/advanced/evaluation_functions/feedback.md @@ -1,5 +1,11 @@ # Base Layer Feedback Implementation +Feedback `cases` are handled by [Shimmy](specification.md#base-layer), not by your function — +Shimmy re-invokes `evaluation_function` once per case. + +This is base-layer behaviour and applies to **every** function regardless of implementation +language; the JSON below is the wire format Shimmy sends, not Python-specific. + Input structure: ```json @@ -20,11 +26,12 @@ Input structure: ``` ## Execution Logic for the `eval` command -1. First `evaluation_function` is called using the response, answer and params -3. If evaluation threw an error, then return the error message -2. If evaluation was successful, check for matching cases - 1. If "params" contains a non-empty list of "cases", determine the correct feedback, add it to the result and return the block (Logic for this is described in the next section) - 2. If "params" doesn't contain a list of cases, simply return the result +1. First `evaluation_function` is called using the response, answer and params. +2. If evaluation threw an error, return the error message. +3. If `params` contains a non-empty list of `cases` and the result is `is_correct: false`, run the case-matching procedure below, merge the outcome into the result and return it. +4. Otherwise, return the result unchanged. + +When a case matches, Shimmy adds `matched_case` (the case's index) to the result, and if that case defines a `mark` (`0` or `1`) it overrides `is_correct`. ## Determining the correct feedback case 1. Iterate through each case in the list of `cases`: diff --git a/docs/advanced/evaluation_functions/local.md b/docs/advanced/evaluation_functions/local.md index 31c6cdee7..3e7a4b098 100644 --- a/docs/advanced/evaluation_functions/local.md +++ b/docs/advanced/evaluation_functions/local.md @@ -1,65 +1,162 @@ # Running and Testing Functions Locally -## Simple +Evaluation functions are developed and tested locally **without** the base-image server: you call +your function directly and run its test suite. The full container — your function behind the +[Shimmy](https://github.com/lambda-feedback/shimmy) base layer — is normally exercised by CI and +in deployment, but you can also [build and run it locally](#testing-against-the-container) to +check the real HTTP API before pushing. +!!! info "This page is about Python functions" + It covers functions built from the current + [`evaluation-function-boilerplate-python`](https://github.com/lambda-feedback/evaluation-function-boilerplate-python), + which uses [Poetry](https://python-poetry.org/) and an `evaluation_function/` package — the + commands below (`poetry`, `pytest`, `python -m evaluation_function.dev`) are all + Python-specific. For Wolfram, Lean or other languages the local loop differs; see + [Other Languages](alternate_languages.md) and the relevant boilerplate's `README.md`. + Functions still on the older AWS Lambda base layer (those with an `app/` directory) are + covered [at the bottom of this page](#older-aws-lambda-base-layer). -## Using Docker [:material-docker:](https://www.docker.com/) -This method builds and runs evaluation functions in the same way they are deployed on AWS as Lambda functions. Extending a pre-built and AWS-maintained [base python image](https://docs.aws.amazon.com/lambda/latest/dg/python-image.html#python-image-base), the container contains a HTTP client which can be used to locally simulate Lambda execution events. +## Run unit tests -Note that this is different from the [simple](#simple) method proposed, in that it gives access to all the functionality provided by the base layer. This means that commands such as `docs` and `healthcheck` can be tested. +Install dependencies and run the test suite with [`pytest`](https://docs.pytest.org/) from the +repository root: -1. Install [Docker](https://docs.docker.com/get-docker/) on your machine +```bash +poetry install +poetry run pytest +``` -2. Navigate to the root directory of your function +This is the same suite the CI pipeline runs on every push and pull request; a function is not +deployed unless it passes. -3. Build the image. This will pull our base image from Dockerhub, extend it with files specific to your evaluation function and name it `eval-tmp`. - ```bash - docker image build -t eval-tmp app - ``` +## Call the function directly -4. Spin up a container using the image built in the previous step. - ```bash - docker run --rm -d --name eval-function -p 9000:8080 eval-tmp - ``` +The boilerplate ships an `evaluation_function/dev.py` helper that calls your `evaluation_function` +directly — the quickest loop while iterating on comparison logic: -5. You can now simulate requests to the function using any request client (like [Insomnia](https://insomnia.rest/) or [Postman](https://www.postman.com/)). By default, the url you can hit is: - ```url - http://localhost:9000/2015-03-31/functions/function/invocations - ``` +```bash +python -m evaluation_function.dev "" "" '' +``` - ???+ warning - *When deployed, our Lambda functions are triggered by calls made through an AWS [API Gateway](https://aws.amazon.com/api-gateway/). This means that when testing locally, events sent should follow the structure of events triggered by that resource. That is, if you want to simulate what it would be like to make web requests to the deployed function.* +For example: - Specifically, this means structuring requests in the following way: - ```json - { - "headers": { - "command": "eval" - }, - "body": { - "response": "a", - "answer": "a", - "params": { - "garlic": "moreish" - } - } - } - ``` +```bash +python -m evaluation_function.dev "2*x" "x + x" '{}' +``` - The main difference is that `headers` and `body` are sent as keys in the main body of the local request. When hitting the deployed function through the API Gateway, the `command` field would instead be passed in the actual HTTP headers of the request - and the actual request body would only contain the `response`, `answer` and `params` fields. +`answer` and the params JSON are optional. See the script's `--help` for its exact arguments, +which vary slightly between functions. -6. *(Optional)* The `run` command specifies the **-d** flag, which spins up the container in detached mode. If you want to inspect the logs of the function, you can run: - ```bash - docker container logs -f eval-function - ``` +## Testing against the container -??? note "Tip" - You will very rarely need this, but you can peek into the running container by opening a shell within it using: +Building the image and sending it real HTTP requests runs the **same container CI builds and +deployment ships**: your function behind [Shimmy](https://github.com/lambda-feedback/shimmy), +serving the API on port `8080`. Use it for the end-to-end checks that calling the function +directly and `pytest` don't cover — schema validation, the µEd and Legacy wire formats, and the +[feedback `cases`](feedback.md) loop. - ```bash - docker exec -it eval-function bash - ``` +!!! info "Applies to any base image" + The steps below use the Python `evaluation_function/` layout for their examples, but the + build and run commands are the same for Wolfram, Lean and `scratch` functions — only the + `Dockerfile` contents differ. See [Other Languages](alternate_languages.md). -## Useful Links +### Build the image -- +From the repository root (where the `Dockerfile` is): + +```bash +docker build -t my-eval-function . +``` + +!!! tip "Podman works too" + [Podman](https://podman.io/) is a drop-in replacement — swap `docker` for `podman` in every + command on this page and the arguments are identical. + +### Run the container + +Expose Shimmy's port `8080`: + +```bash +docker run --rm -p 8080:8080 my-eval-function +``` + +Add `--name my-eval-function` if you want to `docker exec` / `docker cp` into the running +container, and `-e SANDBOX_ENABLED=true` to also exercise the optional +[nsjail](https://github.com/google/nsjail) sandbox that Shimmy applies in production. + +### Health checks + +```bash +curl http://localhost:8080/health +curl --header 'X-Api-Version: 0.1.0' http://localhost:8080/evaluate/health +``` + +`GET /health` is a plain liveness probe; `GET /evaluate/health` is the µEd health route. + +### Send a µEd request + +`POST /evaluate` with an `X-Api-Version: 0.1.0` header — the request the platform sends for +newly registered functions: + +```bash +curl --request POST \ + --url http://localhost:8080/evaluate \ + --header 'Content-Type: application/json' \ + --header 'X-Api-Version: 0.1.0' \ + --data '{ + "submission": { "type": "OTHER", "content": { "value": "x + x" } }, + "task": { "referenceSolution": { "expression": "2*x" } } + }' +``` + +See the [µEd API](specification.md#ed-api) section of the specification for the full +request/response contract. + +### Send a Legacy request + +`POST /` with the command in a `command` header and a bare `response` / `answer` / `params` +body: + +```bash +curl --request POST \ + --url http://localhost:8080/ \ + --header 'Content-Type: application/json' \ + --header 'command: eval' \ + --data '{ "response": "2*x", "answer": "x + x", "params": {} }' +``` + +The response is `{"command": "eval", "result": {...}}`, or `{"error": {"message": ...}}` if the +function raised — see [Legacy API](specification.md#legacy-api). Swapping the header for +`command: healthcheck` runs the function's own test suite inside the container and returns a +pass/fail summary. + +### Postman and other clients + +Any HTTP client works — `curl`, [Insomnia](https://insomnia.rest/), +[Postman](https://www.postman.com/). Point it at the running container: + +- **µEd** — `POST http://localhost:8080/evaluate`, headers `Content-Type: application/json` and + `X-Api-Version: 0.1.0`, body as the µEd JSON above. +- **Legacy** — `POST http://localhost:8080/`, header `Content-Type: application/json` plus a + `command` header (`eval`, `preview` or `healthcheck`), body `{ "response": ..., "answer": ..., + "params": {} }`. + +## Older AWS Lambda base layer + +??? note "Functions not yet migrated" + A small number of functions (for example + [`compareExpressions`](https://github.com/lambda-feedback/compareExpressions)) still extend + the older `ghcr.io/lambda-feedback/baseevalutionfunctionlayer` image and keep the `app/` + directory layout. Their tests run with `python -m unittest app.evaluation_tests`, and the + built image is exercised locally with the AWS + [Runtime Interface Emulator](https://github.com/aws/aws-lambda-runtime-interface-emulator) + (`docker run -p 9000:8080 …`, then POST an API-Gateway-style event to + `http://localhost:9000/2015-03-31/functions/function/invocations`). See the function's own + `README.md` for the details. + +## Useful links + +- [`evaluation-function-boilerplate-python`](https://github.com/lambda-feedback/evaluation-function-boilerplate-python) — template for new Python functions +- [`toolkit-python`](https://github.com/lambda-feedback/toolkit-python) — the `lf_toolkit` helper package +- [`evaluation-function-base`](https://github.com/lambda-feedback/evaluation-function-base) — the base images (Python, Wolfram, Lean, scratch) +- [µEd API specification](https://mued.org/) diff --git a/docs/advanced/evaluation_functions/module.md b/docs/advanced/evaluation_functions/module.md index 5e906a3d9..7fa06f140 100644 --- a/docs/advanced/evaluation_functions/module.md +++ b/docs/advanced/evaluation_functions/module.md @@ -1,17 +1,123 @@ -# evaluation-function-utils Package +# Helper Packages -- Error Reporting -- Schema validation -- Local testing +A **toolkit** implements Shimmy's worker interface so your function only has to provide +comparison logic. Whether one is available depends on the language and the +[base layer](specification.md#base-layer): -## Errors -Submodule containing custom error and exception classes, which can be properly caught by the base evaluation layer, and return more detailed and appropriate errors. +| Language | Toolkit | Used by | Provides | +| --- | --- | --- | --- | +| Python | [`lf_toolkit`](#lf_toolkit) — repo [`toolkit-python`](https://github.com/lambda-feedback/toolkit-python) | Functions on the Shimmy `python` base image | Server wiring, `Result` / `Params` / `Preview`, image upload | +| Python (legacy) | [`evaluation-function-utils`](#evaluation-function-utils-legacy) | Functions on the older AWS Lambda base layer | `EvaluationException`, cross-function client | +| Wolfram Language | [`toolkit-wolfram`](#toolkit-wolfram) | Functions on the Shimmy `wolfram` base image | `ServeEvaluationFunction`, transport wiring, error catching | +| Lean, or any other language | *none yet* (can be provided on request) | Functions on the `lean` / `scratch` base images | — the function talks to Shimmy directly over the [file interface](alternate_languages.md#file) | -### class `EvaluationException` -This class extends the usual python `Exception`, with additional functionality. It can be used to package additional fields and values to errors thrown and returned by evaluation functions. +The Python and Wolfram toolkits are loaded and wired up automatically by their base image. A +Lean or `scratch` function has no toolkit today: it reads the request file and writes the +response file itself — see [Other Languages](alternate_languages.md). If you are building +functions in a language without a toolkit and would benefit from one, the Lambda Feedback team +can provide it on request — [open an issue on `shimmy`](https://github.com/lambda-feedback/shimmy/issues). -!!! example - If at some point in the execution of the [`evaluation_function`](specification.md#the-evaluationfunction), an error is thrown: +## `lf_toolkit` + +`lf_toolkit` (repo [`toolkit-python`](https://github.com/lambda-feedback/toolkit-python)) is +pulled in via the boilerplate's `pyproject.toml` and pre-installed in the +[`evaluation-function-base/python`](https://github.com/lambda-feedback/evaluation-function-base) +image. + +### Server wiring + +`evaluation_function/main.py` connects your function to [Shimmy](specification.md#base-layer): + +```python +from lf_toolkit import create_server, run +from .evaluation import evaluation_function +from .preview import preview_function + + +def main(): + server = create_server() + server.eval(evaluation_function) + server.preview(preview_function) + run(server) + + +if __name__ == "__main__": + main() +``` + +`create_server()` reads the `EVAL_IO` / `EVAL_RPC_TRANSPORT` environment variables that Shimmy +injects and returns the right server (stdio, IPC or file). `healthcheck` is provided by the +toolkit — it discovers and runs the `*_test.py` files — so you do not register it yourself. + +### `Result`, `Params`, `Preview` + +```python +from lf_toolkit.evaluation import Result, Params +from lf_toolkit.preview import Preview + + +def evaluation_function(response, answer, params: Params) -> Result: + return Result(is_correct=response == answer) +``` + +- `Result` — `is_correct`, tagged feedback (`add_feedback(tag, text)`), `response_latex`, + `response_simplified`. Shimmy serialises it (`is_correct`, `feedback`, …). +- `Params` — dict-like wrapper over the request `params`. +- `Preview` — the value returned from `preview_function` (`latex`, `sympy`, `feedback`). + +### Image upload + +`lf_toolkit.evaluation.image_upload` provides `upload_image(...)` and `ImageUploadError` for +functions that return generated images. + +### Errors + +`lf_toolkit` has no structured-exception class. Raising **any** exception from your function +makes Shimmy stop the evaluation and return: + +```json +{ "error": { "message": "" } } +``` + +## `toolkit-wolfram` + +[`toolkit-wolfram`](https://github.com/lambda-feedback/toolkit-wolfram) — the "Evaluation +Function Toolkit for Wolfram" — is the Wolfram-language equivalent of `lf_toolkit`. It is +cloned into the +[`evaluation-function-base/wolfram`](https://github.com/lambda-feedback/evaluation-function-base) +image at a tagged version and loaded by that image's `Bootstrap.wl`. + +A Wolfram function repo does **not** call the toolkit directly. It only provides `evaluate.m` +and `preview.m` defining `` evaluate`EvaluationFunction `` and `` preview`PreviewFunction ``; +the base image's `FUNCTION_COMMAND` / `FUNCTION_ARGS` already point Shimmy at `Bootstrap.wl`, +which loads the toolkit and wires them up. + +For custom wiring or local testing, call +`ServeEvaluationFunction[EvaluationFunction, PreviewFunction]` directly — it reads Shimmy's +`EVAL_IO` / `EVAL_RPC_TRANSPORT` contract and dispatches to whichever transport Shimmy +selected (the file interface, or an RPC transport). A Wolfram error raised by your function is +caught and returned as an error response instead of crashing the worker. See the +[`toolkit-wolfram` README](https://github.com/lambda-feedback/toolkit-wolfram) for the exact +contract and the current list of supported transports. + +## `evaluation-function-utils` (legacy) + +!!! note + This package is only present on the older AWS Lambda base layer. New functions on Shimmy use + [`lf_toolkit`](#lf_toolkit) instead. + +### Errors + +Submodule containing custom error and exception classes, which can be properly caught by the base +evaluation layer, and return more detailed and appropriate errors. + +#### class `EvaluationException` + +This class extends the usual python `Exception`, with additional functionality. It can be used to +package additional fields and values to errors thrown and returned by evaluation functions. + +!!! example + If at some point in the execution of the [`evaluation_function`](specification.md#the-evaluation_function), an error is thrown: ```python from evaluation_function_utils.errors import EvaluationException @@ -39,10 +145,10 @@ This class extends the usual python `Exception`, with additional functionality. This class contains an error_dict property, which packages the additional arguments given to the Exception instance into a JSON-serializable object. It does so in an error-safe way, also reporting serialization errors if they occur. -## Client +### Client This submodule contains a custom `EvaluationFunctionClient`, which can be used to call other deployed evaluation functions. -### class `EvaluationFunctionClient` +#### class `EvaluationFunctionClient` Client wrapped around the botocore.client.Lambda, for invoking deployed evaluation functions. On initialisation, it fetches credentials from environment variables "INVOKER_KEY", "INVOKER_ID" and "INVOKER_REGION", or from an optional environment file prescrived by `env_path`. !!! example @@ -57,4 +163,4 @@ Client wrapped around the botocore.client.Lambda, for invoking deployed evaluati In this example, the evaluation_function completely offloads grading to the deployed 'isExactEqual' function. -*Note:* The `EvaluationFunctionClient.invoke` method was designed to behave exactly as if the [`evaluation_function`](specification.md#the-evaluationfunction) function defined in the targeted deployed function was called directly. This means that if errors are encountered an `EvaluationException` is raised. \ No newline at end of file +*Note:* The `EvaluationFunctionClient.invoke` method was designed to behave exactly as if the [`evaluation_function`](specification.md#the-evaluation_function) function defined in the targeted deployed function was called directly. This means that if errors are encountered an `EvaluationException` is raised. diff --git a/docs/advanced/evaluation_functions/quickstart.md b/docs/advanced/evaluation_functions/quickstart.md index 3b3787fa9..a230f3b68 100644 --- a/docs/advanced/evaluation_functions/quickstart.md +++ b/docs/advanced/evaluation_functions/quickstart.md @@ -4,30 +4,46 @@ It's a cloud function which performs some computation given some user input (the ## Getting Setup for Development +!!! info "These steps are for Python functions" + They assume a function created from the Python boilerplate. The concepts (config, deploy + pipeline, µEd registration) are the same for every language, but the file layout and local + commands in steps 3–4 are Python-specific — for other languages follow + [Other Languages](alternate_languages.md) and the chosen boilerplate's `README.md`. + 1. Get the code on your local machine (Using github desktop or the `git` cli) - - For new functions: create and clone a new repository using the [boilerplate template](https://github.com/lambda-feedback/Evaluation-Function-Boilerplate). **Make sure the new repository is set to public (it needs access to organisation secrets)**. + - For new functions: create a new repository from the [`evaluation-function-boilerplate-python`](https://github.com/lambda-feedback/evaluation-function-boilerplate-python) template via *Use this template*, choosing the `Lambda Feedback` organisation as the owner. **Make sure the new repository is set to public (it needs access to organisation secrets)**. Boilerplates for other languages also exist — [`evaluation-function-boilerplate-wolfram`](https://github.com/lambda-feedback/evaluation-function-boilerplate-wolfram) and [`evaluation-function-boilerplate-lean`](https://github.com/lambda-feedback/evaluation-function-boilerplate-lean); see [Other Languages](alternate_languages.md). - For existing functions: please make your changes on a new separate branch -2. *If you are creating a new function*, you'll need to set it's name (as it will be deployed) in the `config.json` file, available in the root directory. - - The name must be unique. To view existing grading functions, go to: - - [Staging API Gateway Integrations](https://eu-west-2.console.aws.amazon.com/apigateway/main/develop/integrations/attach?api=c1o0u8se7b®ion=eu-west-2&routes=0xsoy4q) - - [Production API Gateway Integrations](https://eu-west-2.console.aws.amazon.com/apigateway/main/develop/integrations/attach?api=cttolq2oph&integration=qpbgva8®ion=eu-west-2&routes=0xsoy4q) -3. You are now ready to start making changes and implementing features by editing each of the three main function-logic files: - 1. **`app/evaluation.py`**: This file contains the main `evaluation_function` function, which ultimately gets called to compare a *response* to an *answer*. +2. *If you are creating a new function*, set its deployed name in the `config.json` file in the root directory: + + ```json + { "EvaluationFunctionName": "myFunction" } + ``` + + The name must be unique across the organisation and is conventionally `lowerCamelCase`. +3. You are now ready to start making changes. The function logic lives in the `evaluation_function/` package: + 1. **`evaluation_function/evaluation.py`**: contains the main `evaluation_function`, which is called to compare a *response* to an *answer*. [`evaluation.py` Specification](specification.md#evaluationpy){ .md-button } - 2. **`app/evaluation_tests.py`**: This is where you can test the logic in `evaluation.py`, following the standard `unittest` format. + 2. **`evaluation_function/preview.py`**: contains `preview_function`, which pre-processes a *response* for live display (e.g. rendered LaTeX) without grading it. + + 3. **`evaluation_function/evaluation_test.py`**: where you test the logic in `evaluation.py`, using [`pytest`](https://docs.pytest.org/). - [`evaluation_tests.py` Specification](specification.md#evaluation_testspy){ .md-button } + [`evaluation_test.py` Specification](specification.md#evaluation_testpy){ .md-button } - 3. Documentation files: - - **`app/docs/dev.md`**: This file should be edited to reflect any changes/features implemented, following a developer perspective. It is baked into the function's image to be pulled by this documentation website under the [deployed functions](index.md) section. - - - **`app/docs/user.md`**: This file documents how the function can be used by a teacher user, from the perspective of editing content on the [LambdaFeedback]({{ urls.client }}) platform. This time, files are collated and displayed in the [Teacher](../../teacher/index.md) section. + 4. **`evaluation_function/main.py`**: the entry point. It calls `lf_toolkit.create_server()` and registers your `evaluation_function` and `preview_function` with it. You rarely need to change this file. -4. Changes can be tested locally by running the tests you've written using: + 5. Documentation files: + - **`docs/dev.md`**: edited to reflect any changes/features from a developer perspective. It is baked into the function's image and pulled into this site under the [deployed functions](index.md) section. + + - **`docs/user.md`**: documents how a teacher uses the function when editing content on the [LambdaFeedback]({{ urls.client }}) platform. These files are displayed in the [Teacher](../../teacher/index.md) section. + + - **`README.md`**: replace the boilerplate's generic title, description and Quickstart section with your function's own, and have it link to `docs/dev.md`, `docs/user.md` and this site rather than restate them. See [the README convention](specification.md#function-repository-readmemd). + +4. Changes can be tested locally by running your tests from the repository root: ```bash -python -m unittest app/evaluation_tests.py +poetry install +poetry run pytest ``` [Running and Testing Functions Locally](local.md){ .md-button } @@ -42,16 +58,14 @@ python -m unittest app/evaluation_tests.py !!! note The build and deploy steps are implemented as reusable workflows maintained in [lambda-feedback/evaluation-function-workflows](https://github.com/lambda-feedback/evaluation-function-workflows). -6. You can now test the deployed evaluation function using your prefered request client (such as [Insomnia](https://insomnia.rest/) or [Postman](https://www.postman.com/) or simply `curl` from a terminal). Functions are made available at: - ```url - https://c1o0u8se7b.execute-api.eu-west-2.amazonaws.com/default/ - ``` +6. Once the deploy workflow has run, the platform hosts your function at a public URL. You can find it in the [Admin Panel]({{ urls.client }}admin/functions) after registering the function (next step), and test it with any request client (`curl`, [Insomnia](https://insomnia.rest/), [Postman](https://www.postman.com/)). - !!! example "Example µEd Request to SymbolicEqual" + !!! example "Example µEd request" ```bash curl --request POST \ - --url https://c1o0u8se7b.execute-api.eu-west-2.amazonaws.com/default/symbolicEqual/evaluate \ + --url https:///evaluate \ --header 'Content-Type: application/json' \ + --header 'X-Api-Version: 0.1.0' \ --data '{ "submission": { "type": "MATH", "content": { "expression": "x + x" } }, "task": { "referenceSolution": { "expression": "2*x" } } @@ -60,19 +74,18 @@ python -m unittest app/evaluation_tests.py See the [µEd API](specification.md#ed-api) section of the specification for full request/response details. Functions still running the **Legacy** API instead use the `command` header — see [Legacy API](specification.md#legacy-api). -7. In order to make your new function available on the LambdaFeedback platform, you have to register it via the [Admin Panel]({{ urls.client }}admin/functions). This is done by supplying its name, url (the same as the one above) and supported response types. +7. To make your new function available on the LambdaFeedback platform, register it via the [Admin Panel]({{ urls.client }}admin/functions) by supplying its name, URL and supported response types. !!! note - New evaluation functions should be registered as **µEd** (a standard, path-based API — see [Chat Functions](../chat_functions/quickstart.md) for a general introduction to µEd on Lambda Feedback, and [mued.org](https://mued.org/) for the specification). The **Legacy** command-header API documented on this page is being phased out — only a small number of functions that haven't yet migrated still use it. + New evaluation functions should be registered as **µEd** (a standard, path-based API — see [Chat Functions](../chat_functions/quickstart.md) for a general introduction to µEd on Lambda Feedback, and [mued.org](https://mued.org/) for the specification). The **Legacy** command-header API — described in the [specification](specification.md#legacy-api) — is frozen and no longer developed, but Shimmy still serves it. ## More Info - [General Function Specification and Behaviour](specification.md) - Function philosophy including deployment strategy - Request/Response schemas and communication spec - - Base layer logic, properties and behaviour + - Base layer (Shimmy) logic, properties and behaviour -- [EvaluationFunctionUtils](module.md) (python package) - - Error Reporting - - Schema validation - - Local testing \ No newline at end of file +- [Helper packages](module.md) + - `lf_toolkit` — server wiring, `Result` / `Params` / `Preview`, image upload + - `evaluation-function-utils` — the legacy package (error reporting, cross-function client) \ No newline at end of file diff --git a/docs/advanced/evaluation_functions/specification.md b/docs/advanced/evaluation_functions/specification.md index 9b33b1edd..297f6418b 100644 --- a/docs/advanced/evaluation_functions/specification.md +++ b/docs/advanced/evaluation_functions/specification.md @@ -1,6 +1,20 @@ # Evaluation Function Specification -## Introduction and Philosophy +This page has three parts: + +- **[Universal specification](#universal-specification)** — the request/response contract, the + APIs, the base layer and the documentation layout. Every evaluation function follows this, + regardless of the language it is written in. +- **[Python specification](#python-specification)** — the file layout, `lf_toolkit` wiring and + test setup for functions built from + [`evaluation-function-boilerplate-python`](https://github.com/lambda-feedback/evaluation-function-boilerplate-python). +- **[Wolfram specification](#wolfram-specification)** — the equivalent for Wolfram-language + functions built on `toolkit-wolfram`. + +Functions in any other language (Lean, or the bare `scratch` image) follow the universal +specification plus their own boilerplate — see [Other languages](#other-languages). + +## Universal specification Functionality for each evaluation function is split up as follows: @@ -8,7 +22,7 @@ Functionality for each evaluation function is split up as follows: Universal function behaviour applicable to _every_ function, such as the ability to run tests, return documentation and execute the evaluation is handled by the [**Base Layer**](#base-layer). This is the docker image which is extended by every developed evaluation function. !!! abstract "" -Functionality that may be required in more than one function (but not necessarily all), such as the ability to call already deployed functions and error reporting is handled by the [**evaluation_function_utils**](module.md) python package. This package comes pre-installed in the base layer, and can optionally be imported and called from the _evaluation_function_. +Functionality that several functions need but not all — a `Result` / `Params` API, image upload, structured errors, calling other deployed functions — is provided by a **language-specific helper package**. Python functions on the Shimmy base layer use [**`lf_toolkit`**](module.md), which is pulled in by the boilerplate and pre-installed in the Python base image; functions still on the older AWS Lambda base layer use the legacy [**`evaluation-function-utils`**](module.md) package. Wolfram functions use [**`toolkit-wolfram`**](module.md#toolkit-wolfram). Other languages have no toolkit yet (see [Other languages](#other-languages)). !!! info "" Finally, specific comparison logic and handling of bespoke evaluation parameters is done in the custom [**evaluation_function**](#the-evaluation_function), unique to each deployed instance. This is the logic that differenciates each function (comparing numbers, matrices, images, equations, graphs, text, tables, etc ...). @@ -16,163 +30,13 @@ Finally, specific comparison logic and handling of bespoke evaluation parameters !!! note "" New evaluation functions should use the [**µEd API**](#ed-api). The [**Legacy API**](#legacy-api) is being phased out — only a small number of functions that haven't yet migrated still use it. -## µEd API - -Evaluation functions can be registered to serve the [µEd API](https://mued.org/) — a standard, path-based request/response format shared with [Chat Functions](../chat_functions/quickstart.md). Requests are routed and validated by the [base layer](#base-layer) against the [µEd OpenAPI specification](https://github.com/lambda-feedback/BaseEvalutionFunctionLayer/blob/main/schemas/muEd/openapi-v0_1_0.yml); only `POST /evaluate` and `GET /evaluate/health` are implemented for evaluation functions. - -Importantly, **the µEd routes call the same [`evaluation_function`](#the-evaluation_function) and `preview_function` you write for the Legacy API** — the base layer translates between the two wire formats, so no separate implementation is needed to support both. - -### `POST /evaluate` - -Runs an evaluation and returns feedback for a submission. If `preSubmissionFeedback.enabled` is `true` in the request, a non-final preview is returned instead (see [Pre-submission feedback](#pre-submission-feedback) below). - -!!! example - ```bash - curl --request POST \ - --url https://c1o0u8se7b.execute-api.eu-west-2.amazonaws.com/default/isExactEqual/evaluate \ - --header 'Content-Type: application/json' \ - --data '{ - "submission": { - "type": "OTHER", - "content": { "value": "x + x" } - }, - "task": { - "referenceSolution": { "expression": "2*x" } - } - }' - ``` - - - -## Legacy API - -Commands are handled by the [base layer](#base-layer). They define a unified interface for interacting with all deployed evaluation functions on the web. Practically, these are specified in the "command" request header. - -!!! example - To execute the `docs-user` command for a function, the following header would be specified alonside the http request made to the endpoint on which the function is made available: - - ```bash - curl --request GET \ - --url https://c1o0u8se7b.execute-api.eu-west-2.amazonaws.com/default/isExactEqual \ - --header 'command: docs-user' - ``` - -### `eval` - -This is the default command, used to compare a student's `response` and correct `answer`, given certain `params`. Outputs for this command depend on the success of the execution of the user-defined [`evaluation_function`](#the-evaluation_function). If an error was thrown during execution, it is caught by the main handler and an error block is returned - otherwise, successful execution outputs are supplied under a `result` field. - -!!! success "Output Structure: Successful evaluation" - - ``` { .python .annotate } - { - "command": "eval", - "result": { - "is_correct": "", - - # Optional fields added by feedback generation (1) - "feedback": "", - "warnings": "" - - # This output can also contain any number of fields given by `evaluation_function` - } - } - ``` - - 1. See the [Feedback Page](feedback.md) for more information - -!!! fail "Output Structure: Error thrown during Execution" - - ``` { .python .annotate } - { - "command": "eval", - "error": { - "message": "", # Always present - - # This object can contain other number of additional fields - # passed through by the EvaluationException (1) for debugging e.g.: - "serialization_errors": [], - "culprit": "user", - "detail": "..." - } - } - ``` - - 1. This is a custom error class from the [evaluation-function-utils](module.md) package, which developers are encouraged to use in order to output richer errors. See the [Error handling](#error-handling) section for more information. - -### `preview` - -This command is similar to `eval`, except it doesn't return whether an answer is correct or provide feedback. Instead, `preview` provides a way for students view their response after some pre-processing, e.g. as rendered LaTeX when using Sympy for symbolic algebra. - -This should be faster to compute than `eval`, allowing students to get live preview of their response. - -### `healthcheck` - -This command runs and returns a summary three testing suites: requests, responses and evaluation. Request and response tests check that inputs and outputs to the function work correctly, and follow the correct syntax. Evaluation tests are unique to each evaluation function and test the actual comparison logic. - -### `docs-user` - -Command returns the `docs/user.md` file (base64 encoded) - -### `docs-dev` - -Command returns the `docs/dev.md` file (base64 encoded) - -## Base Layer - -## File Structure - -A standard evaluation function repository based on the provided [boilerplate](https://github.com/lambda-feedback/Evaluation-Function-Boilerplate) will have the following file structure: - -```bash -app/ - __init__.py - evaluation.py # Script containing the main evaluation_function - evaluation_tests.py # Unittests for the main evaluation_function - requirements.txt # list of packages needed for algorithm.py - Dockerfile # for building whole image to deploy to AWS - - docs/ # Documentation pages for this function (required) - dev.md # Developer-oriented documentation - user.md # LambdaFeedback content author documentation - -.github/ - workflows/ - staging-deploy.yml # Test, lint and deploy to staging on push to main - production-deploy.yml # Manually-triggered deploy to production - test-lint.yml # Test and lint on pull requests - -config.json # Specify the name of the evaluation function in this file -README.md -.gitignore -``` - -!!! note - The `staging-deploy.yml` and `production-deploy.yml` workflows call into reusable workflows maintained in [lambda-feedback/evaluation-function-workflows](https://github.com/lambda-feedback/evaluation-function-workflows), which handle the actual build and deploy steps. - -!!! warning - - If you want to split up function logic into different files, these must be added to the `Dockerfile`. This is so they are packaged with the built image when deployed. For example, if `evaluation.py` imports functionality from an `app/utils.py` file, then the following line must be added: - - ```dockerfile linenums="9" hl_lines="7 8" - RUN pip3 install -r requirements.txt - - # Copy the evaluation and testing scripts - COPY evaluation.py ./app/ - COPY evaluation_tests.py ./app/ - - # Copy additional files - COPY utils.py ./app/ - - # Copy Documentation - COPY docs/dev.md ./app/docs/dev.md - ``` - -## `evaluation.py` - -The entire framework, validation and testing developed around evaluation functions is ultimately used to get to this file, or the `evaluation_function` function within it, to be more precise. - ### The `evaluation_function` +Every function implements an `evaluation_function` (and, optionally, a `preview_function`). Both +the [µEd API](#ed-api) and the [Legacy API](#legacy-api) routes call the same function — the +base layer translates between the wire formats — so there is only ever one implementation to +write. + #### Inputs All evaluation functions are passed three arguments: @@ -234,20 +98,27 @@ When a student submits a response to a response area the number of previously su #### Outputs -The function should output a single JSON-encodable dictionary. Although a large amount of freedom is given to what this dict contains, when utilising the function alongside the [lambdafeedback](https://lambdafeedback.com/) web app, a few values are expected/able to be consumed: +The function returns a JSON-encodable result (Python functions can return an +[`lf_toolkit`](module.md) `Result` object, which the base layer serialises; a plain dictionary +also works). Although a large amount of freedom is given to what the result contains, when +utilising the function alongside the [lambdafeedback](https://lambdafeedback.com/) web app, a +few values are expected/able to be consumed: **`is_correct: `**: Boolean parameter indicate whether the comparison between `response` and `answer` was deemed correct under the parameters. This field is then used by the web app to provide the most simple feedback to the user (green/red). !!! info _More standardised function outputs that the frontend can consume are to come_ -### Error Handling +#### Error Handling -Error reporting should follow a specific approach for all evaluation functions. **If the `evaluation_function` you've written doesn't throw any errors, then it's output is returned under the `result` field - and assumed to have worked properly**. This means that if you catch an error in your code manually, and simply return it - the frontend will assume everything went fine. Instead, errors can be handled in two ways: +Error reporting should follow a specific approach for all evaluation functions. **If the `evaluation_function` you've written doesn't throw any errors, then it's output is returned under the `result` field - and assumed to have worked properly**. This means that if you catch an error in your code manually, and simply return it - the frontend will assume everything went fine. Instead, errors should be signalled by failing, not by returning an `error` field. -**Letting `evaluation_function` fail**: On the request handler in the [Base Layer](#base-layer), the call to evaluation_function is wrapped in a try/except which catches any exception. This causes the evaluation to stop completely, returning a standard message, and a repr of the exception thrown in the `error.detail` field. +**Letting `evaluation_function` fail**: [Shimmy](#base-layer) wraps the call to `evaluation_function` in a try/except which catches any exception. This causes the evaluation to stop completely and return `{"error": {"message": ""}}`. -**Custom errors**: If you want to report more detailed errors from your function, use the `EvaluationException` class provided in the [evaluation-function-utils](module.md#errors) package. These are caught before all other standard exceptions, and are dealt with in a different way. These provide a way for your function to throw errors and stop executing safely, while supplying more accurate feedback to the front-end. +**Custom errors**: functions on the older AWS Lambda base layer can raise the `EvaluationException` class from the [evaluation-function-utils](module.md#class-evaluationexception) package to attach extra fields to the error block. These are caught before all other standard exceptions and dealt with differently, letting the function stop safely while supplying richer feedback to the front-end. + +!!! note + `EvaluationException` is part of the legacy `evaluation-function-utils` package. Functions built on Shimmy with `lf_toolkit` have no structured-error equivalent yet — raising **any** exception produces the `{"error": {"message": ...}}` block above. !!! Example It is discouraged to do the following in the evaluation code: @@ -261,7 +132,7 @@ It is discouraged to do the following in the evaluation code: } ` - As this causes the actual function output (by the AWS lambda function) to be: + As this causes the actual function output to be: ```json { "command": "eval", @@ -274,7 +145,7 @@ It is discouraged to do the following in the evaluation code: } ``` - Instead, use custom exceptions from the [evaluation-function-utils](module.md#errors) package. + Instead, use custom exceptions from the [evaluation-function-utils](module.md#class-evaluationexception) package. ```python if something.bad.happened(): raise EvaluationException(message="Some important message", other='details') @@ -293,47 +164,114 @@ It is discouraged to do the following in the evaluation code: This immediately indicates to the frontend client that something has gone wrong, allowing for proper feedback to be displayed. -## `evaluation_tests.py` +### µEd API -This file is intended to contain unit tests for the `evaluation_function`. Python's built-in -[`unittest`](https://docs.python.org/3/library/unittest.html) framework is used. -These tests are run by Github Actions whenever changes are pushed to the main branch, and -the evaluation function is not deployed unless all the tests pass. +Evaluation functions can be registered to serve the [µEd API](https://mued.org/) — a standard, path-based request/response format shared with [Chat Functions](../chat_functions/quickstart.md). Requests are routed and validated by the [base layer](#base-layer) against the [µEd OpenAPI specification](https://github.com/lambda-feedback/shimmy/blob/main/runtime/schema/mued_v0.1.0.yml); only `POST /evaluate` and `GET /evaluate/health` are implemented for evaluation functions. An optional `X-Api-Version: 0.1.0` header selects the schema version. -!!! Example -A minimal example of a test: -```python -import unittest -from .evaluation import evaluation_function - -# Tests are functions beginning with "test_" in -# a class that inherits from unittest.TestCase -class TestEvaluationFunction(unittest.TestCase): - def test_trivial(self): - result = evaluation_function("a + b", "a + b", {}) - self.assertTrue(result["is_correct"]) -``` -Tests can be run locally using -```bash -$ python -m unittest app.evaluation_tests -``` +Importantly, **the µEd routes call the same [`evaluation_function`](#the-evaluation_function) and `preview_function` you write for the Legacy API** — the base layer translates between the two wire formats, so no separate implementation is needed to support both. -### Autotests +#### `POST /evaluate` -For writing simple tests, it may be easier to write the tests in a config file and have them -run on the evaluation function automatically. This can be achieved using the autotests library, -which can easily be integrated into an existing project by adding a decorator to the test class. -See the autotests [README](https://github.com/lambda-feedback/evaluation-function-auto-tests) -for more information. +Runs an evaluation and returns feedback for a submission. If `preSubmissionFeedback.enabled` is `true` in the request, a non-final preview is returned instead (equivalent to the Legacy [`preview`](#preview) command). -Another benefit of this approach is that the tool that collects evaluation function documentation -([EvalDocsLoader](https://github.com/lambda-feedback/EvalDocsLoader)) can read this file and -auto-generate examples of correct and incorrect responses. This can help new users understand -the capabilities of your evaluation function. +!!! example + ```bash + curl --request POST \ + --url https:///evaluate \ + --header 'Content-Type: application/json' \ + --header 'X-Api-Version: 0.1.0' \ + --data '{ + "submission": { + "type": "OTHER", + "content": { "value": "x + x" } + }, + "task": { + "referenceSolution": { "expression": "2*x" } + } + }' + ``` -For an example of how this looks, see the user docs for [compareBoolean](https://lambda-feedback.github.io/user-documentation/user_eval_function_docs/compareBoolean/#examples-from-integration-tests). +### Legacy API + +The Legacy API is the original command-based interface. It is **frozen** — no longer extended — but [Shimmy](#base-layer) still serves it, so functions do not need to migrate to keep working. It is exposed at `POST /`, with the command given in a request header named `command` (`eval` if the header is absent). The request body is a bare JSON object (`response`, `answer`, `params` — no wrapper); the response is `{"command": ..., "result": {...}}`, or `{"error": {"message": ...}}` if the function raised. + +!!! example + To run the `eval` command against a deployed function: + + ```bash + curl --request POST \ + --url https:/// \ + --header 'Content-Type: application/json' \ + --header 'command: eval' \ + --data '{ "response": "2*x", "answer": "x + x", "params": {} }' + ``` + +#### `eval` + +This is the default command, used to compare a student's `response` and correct `answer`, given certain `params`. Outputs for this command depend on the success of the execution of the user-defined [`evaluation_function`](#the-evaluation_function). If an error was thrown during execution, it is caught by the main handler and an error block is returned - otherwise, successful execution outputs are supplied under a `result` field. + +!!! success "Output Structure: Successful evaluation" + + ``` { .python .annotate } + { + "command": "eval", + "result": { + "is_correct": "", + + # Optional fields added by feedback generation (1) + "feedback": "", + "warnings": "" + + # This output can also contain any number of fields given by `evaluation_function` + } + } + ``` + + 1. See the [Feedback Page](feedback.md) for more information + +!!! fail "Output Structure: Error thrown during Execution" + + ``` { .python .annotate } + { + "command": "eval", + "error": { + "message": "", # Always present + + # This object can contain other number of additional fields + # passed through by the EvaluationException (1) for debugging e.g.: + "serialization_errors": [], + "culprit": "user", + "detail": "..." + } + } + ``` + + 1. This is a custom error class from the [evaluation-function-utils](module.md) package, which developers are encouraged to use in order to output richer errors. See the [Error handling](#error-handling) section for more information. -## Documentation +#### `preview` + +This command is similar to `eval`, except it doesn't return whether an answer is correct or provide feedback. Instead, `preview` provides a way for students view their response after some pre-processing, e.g. as rendered LaTeX when using Sympy for symbolic algebra. + +This should be faster to compute than `eval`, allowing students to get live preview of their response. + +#### `healthcheck` + +Runs the function's own test suite (test discovery over the `*_test.py` files) and returns a summary: `{"tests_passed": , "successes": [...], "failures": [...], "errors": [...]}`. + +### Base Layer + +The base layer is [**Shimmy**](https://github.com/lambda-feedback/shimmy), an HTTP server bundled into the [`evaluation-function-base`](https://github.com/lambda-feedback/evaluation-function-base) image that every function extends. It provides the behaviour common to all functions, so the function itself only implements comparison logic. Shimmy: + +- serves the [µEd API](#ed-api) (`POST /evaluate`, `GET /evaluate/health`) and the [Legacy API](#legacy-api) (`POST /`, command in a header), plus a `GET /health` liveness probe, all on port `8080`; +- validates each request against the relevant schema before your code runs; +- launches your function as a child process and talks to it over JSON-RPC — Python functions use the [`lf_toolkit`](module.md) package for this — or, for other languages, a file-based interface (see [Other Languages](alternate_languages.md)); +- runs the [feedback `cases`](feedback.md) loop, re-invoking your function once per case; +- optionally sandboxes the function with [nsjail](https://github.com/google/nsjail) (`SANDBOX_ENABLED=true`). + +!!! note "Older base layer" + Functions that have not yet migrated extend [`BaseEvalutionFunctionLayer`](https://github.com/lambda-feedback/BaseEvalutionFunctionLayer) instead — an Amazon Linux image built on the AWS Lambda runtime. It serves the Legacy API only (including `docs-user` / `docs-dev`) and is tested locally with the AWS Runtime Interface Emulator; see [Running Functions Locally](local.md#older-aws-lambda-base-layer). + +### Documentation Evaluation function documentation is stored in two files, which contain documentation for developers and users respectively. These files are fetched by @@ -348,11 +286,150 @@ In order for EvalDocsLoader to find your docs, your evaluation function must: Once these requirements are met, the docs you write should appear on the documentation site. -### `docs/dev.md` +#### `docs/dev.md` This should contain documentation that would be useful for new developers working on your function. -### `docs/user.md` +#### `docs/user.md` This should contain information for non-technical users, such as an overview of capabilities, examples, and a description of parameters. + +#### Function repository `README.md` + +Every boilerplate ships a generic `README.md` that documents the *template* itself. When you +create a function from it, that `README.md` should be made specific to your function: + +1. replace the title and description with your function's purpose; +2. delete the boilerplate "Quickstart" / template-setup section; +3. keep the developer- and user-facing documentation in `docs/dev.md` and `docs/user.md` (these + are what EvalDocsLoader publishes to this site), and have the `README.md` **link** to them + and to this page rather than restate their content. + +This keeps a single source of truth: behaviour shared by all functions is documented here, +function-specific behaviour lives in that function's `docs/`, and the `README.md` only points +at both. A generic, unmodified `README.md` is a sign the function still needs this step. + +## Python specification + +Describes functions built from +[`evaluation-function-boilerplate-python`](https://github.com/lambda-feedback/evaluation-function-boilerplate-python), +which uses [Poetry](https://python-poetry.org/) and an `evaluation_function/` package. See +[Running and Testing Functions Locally](local.md) for the local workflow. + +### File Structure + +A function created from [`evaluation-function-boilerplate-python`](https://github.com/lambda-feedback/evaluation-function-boilerplate-python) has this layout: + +```bash +evaluation_function/ + __init__.py + main.py # Entry point: create_server() + register eval/preview (rarely edited) + evaluation.py # The main evaluation_function + preview.py # The preview_function + evaluation_test.py # pytest tests for evaluation_function + preview_test.py # pytest tests for preview_function + dev.py # Local CLI: python -m evaluation_function.dev + +docs/ # Documentation pages for this function (required) + dev.md # Developer-oriented documentation + user.md # LambdaFeedback content-author documentation + +.github/ + workflows/ # Reusable CI/CD from lambda-feedback/evaluation-function-workflows + +config.json # { "EvaluationFunctionName": "" } +Dockerfile +pyproject.toml # Dependencies (Poetry); lf_toolkit is pulled in here +poetry.lock +README.md +``` + +The `Dockerfile` extends the base image and tells Shimmy how to start the worker: + +```dockerfile +FROM ghcr.io/lambda-feedback/evaluation-function-base/python:3.12 +# ... poetry install ... +COPY evaluation_function ./evaluation_function +ENV FUNCTION_COMMAND="python" +ENV FUNCTION_ARGS="-m,evaluation_function.main" +ENV FUNCTION_RPC_TRANSPORT="ipc" +``` + +Extra modules you add under `evaluation_function/` are picked up by the existing `COPY evaluation_function ./evaluation_function` line, so splitting logic across files needs no Dockerfile change. + +!!! note + The `staging-deploy.yml` and `production-deploy.yml` workflows call into reusable workflows maintained in [lambda-feedback/evaluation-function-workflows](https://github.com/lambda-feedback/evaluation-function-workflows), which handle the actual build and deploy steps. + +!!! note "Older `app/` layout" + Functions on the older AWS Lambda base layer use an `app/` directory holding `evaluation.py`, `evaluation_tests.py`, `requirements.txt`, a `Dockerfile` and `docs/`, with `config.json` and the workflows at the repository root. There, each additional source file must be added to the `Dockerfile` with its own `COPY` line. + +### `evaluation.py` + +The entire framework, validation and testing developed around evaluation functions is ultimately used to get to `evaluation_function/evaluation.py`, or the `evaluation_function` within it, to be more precise. `evaluation_function/main.py` registers it with the base layer via [`lf_toolkit`](module.md); you normally only edit `evaluation.py` (and `preview.py`). The arguments and return value are the [universal `evaluation_function` contract](#the-evaluation_function) above; `lf_toolkit` provides `Result` / `Params` / `Preview` wrappers for it (see [Helper Packages](module.md#lf_toolkit)). + +### `evaluation_test.py` + +This file contains the tests for `evaluation_function`, run with [`pytest`](https://docs.pytest.org/). +Github Actions runs them on every push and pull request, and the function is not deployed unless +they pass. + +!!! Example + A minimal test: + ```python + from .evaluation import evaluation_function + + def test_trivial(): + result = evaluation_function("a + b", "a + b", {}) + assert result.is_correct + ``` +Run them locally from the repository root with: +```bash +poetry run pytest +``` + +#### Autotests + +For writing simple tests, it may be easier to write the tests in a config file and have them +run on the evaluation function automatically. This can be achieved using the autotests library, +which can easily be integrated into an existing project by adding a decorator to the test class. +See the autotests [README](https://github.com/lambda-feedback/evaluation-function-auto-tests) +for more information. + +Another benefit of this approach is that the tool that collects evaluation function documentation +([EvalDocsLoader](https://github.com/lambda-feedback/EvalDocsLoader)) can read this file and +auto-generate examples of correct and incorrect responses. This can help new users understand +the capabilities of your evaluation function. + +For an example of how this looks, see the user docs for [compareBoolean](https://lambda-feedback.github.io/user-documentation/user_eval_function_docs/compareBoolean/#examples-from-integration-tests). + +## Wolfram specification + +Wolfram-language functions extend the +[`evaluation-function-base/wolfram`](https://github.com/lambda-feedback/evaluation-function-base) +image, which bundles [`toolkit-wolfram`](module.md#toolkit-wolfram) (the "Evaluation Function +Toolkit for Wolfram") — the Wolfram equivalent of `lf_toolkit`. + +Start from +[`evaluation-function-boilerplate-wolfram`](https://github.com/lambda-feedback/evaluation-function-boilerplate-wolfram). +Your function defines an evaluation function and a preview function whose return value is an +association containing `is_correct`, `feedback` and `error` (`Null` on success) — the Wolfram +form of the [universal contract](#the-evaluation_function) above. The toolkit reads Shimmy's +environment contract and dispatches each request to your function (via +`ServeEvaluationFunction`), so the function never handles the wire format itself; a Wolfram +error it raises is caught and returned as an error response. + +For the exact entry-point contract, the `Dockerfile` settings +(`FUNCTION_COMMAND` / `FUNCTION_ARGS` / `FUNCTION_INTERFACE`) and the transports the toolkit +currently supports, see the +[`toolkit-wolfram` README](https://github.com/lambda-feedback/toolkit-wolfram) and +[Other Languages](alternate_languages.md). + +## Other languages + +Lean functions, and functions on the bare `scratch` base image, follow the +[universal specification](#universal-specification) above and talk to Shimmy over the file +interface — one process per request, reading a request JSON file and writing a response JSON +file. There is no helper toolkit for these yet (one can be provided on request). See +[Other Languages](alternate_languages.md) for the worker interfaces, the `Dockerfile` +environment variables, and the per-language boilerplates. diff --git a/docs/teacher/reference/evaluation_functions/index.md b/docs/teacher/reference/evaluation_functions/index.md index dfefb6167..97c7b5acf 100644 --- a/docs/teacher/reference/evaluation_functions/index.md +++ b/docs/teacher/reference/evaluation_functions/index.md @@ -1,3 +1,3 @@ # Evaluation Functions -Evaluation functions are responsible for taking in a user's response, comparing it with a correct answer, and providing feedback to the frontend application. Living as containerized Lambda functions on the cloud, they are infinitely customisable and language-agnostic. Content authors should be able to create their own at will. However, we are aware that in a lot of cases, this grading logic will be similar, which is why a few functions have already been created. +Evaluation functions are responsible for taking in a user's response, comparing it with a correct answer, and providing feedback to the frontend application. Living as containerised microservices on the cloud, they are infinitely customisable and language-agnostic. Content authors should be able to create their own at will. However, we are aware that in a lot of cases, this grading logic will be similar, which is why a few functions have already been created. diff --git a/mkdocs.yml b/mkdocs.yml index d5d2ecdb8..1a9d78f5d 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -80,10 +80,10 @@ nav: - General Specification: "advanced/evaluation_functions/specification.md" - Feedback: "advanced/evaluation_functions/feedback.md" - Testing Functions Locally: "advanced/evaluation_functions/local.md" - - Evaluation Function Utils: "advanced/evaluation_functions/module.md" + - Helper Packages: "advanced/evaluation_functions/module.md" - Deployed Functions: - "advanced/evaluation_functions/index.md" - - Alternate Function Languages: "advanced/evaluation_functions/alternate_languages.md" + - Other Languages: "advanced/evaluation_functions/alternate_languages.md" - Chat functions: - Quickstart: "advanced/chat_functions/quickstart.md" - Testing Functions Locally: "advanced/chat_functions/local.md"