docs: restructure how-to and tutorial pages

docs/explanations/extension-code.md

@@ -11,7 +11,7 @@ Plans and devices can be in any pip-installable package, such as:
 
 * A package on pypi
 * A GitHub repository
-* A local directory via the [scratch area](../how-to/edit-live.md).
+* A local directory via the [scratch area](../how-to/1.5%20edit-live.md).
 
 The easiest place to put the code is a repository created with the [`python-copier-template`](https://diamondlightsource.github.io/python-copier-template/main/index.html). Which can then become any of the above. [Example for the I22 beamline](https://github.com/DiamondLightSource/i22-bluesky).
 

docs/how-to/configure-app.md → docs/how-to/1. configure-app.md

@@ -1,4 +1,4 @@
-# Configure the Application
+# 1. Configure the Application
 
 Blueapi's default configuration can be overridden
 by defining a yaml file which can be passed to the `blueapi` command.
@@ -16,3 +16,12 @@ avoid it having to be passed to every time the cli is used:
 export BLUEAPI_CONFIG=/path/to/file.yaml
 blueapi <subcommand>
 ```
+
+The next few pages detail various aspects that can be configured.
+
+1.1 [Add plans and devices](./1.1%20add-plans-and-devices.md)
+1.2 [Authentication/Authorization](./1.2%20authn-authz.md)
+1.3 [Logging](./1.3%20configure-logging.md)
+1.4 [Numtracker](./1.4%20integrate-with-numtracker.md)
+1.5 [Install editable Python packages](./1.5%20edit-live.md)
+1.6 [Tiled](./1.6%20configure-tiled.md)

docs/how-to/add-plans-and-devices.md → docs/how-to/1.1 add-plans-and-devices.md

@@ -1,16 +1,16 @@
-# Add Plans and Devices to your Blueapi Environment
+# 1.1 Add Plans and Devices to your Blueapi Environment
 
 :::{seealso}
 [The bluesky documentation](https://blueskyproject.io/bluesky/main/index.html) for an introduction to the nature of plans and devices and why you would want to customize them for your experimental needs.
 :::
 
-Blueapi can be configured to load custom code at startup that defines plans and devices. The code must be in your Python environment (via `pip install <package>`) or your [scratch area](./edit-live.md).
+Blueapi can be configured to load custom code at startup that defines plans and devices. The code must be in your Python environment (via `pip install <package>`) or your [scratch area](./1.5%20edit-live.md).
 
 
 ## Configuration
 
 :::{seealso}
-[Configure the Application](./configure-app.md)
+[Configure the Application](./1.%20configure-app.md)
 :::
 
 First determine the import path of your code. If you were going to import it in a Python file, what would you put?
@@ -25,6 +25,8 @@ To add plans, you would add the following into your configuration file:
 :language: yaml
 ```
 
+For BlueAPI to detect plans from repositories and be registered, they must be added to the `__init__.py` file with `__all__` = [<plan_name>, ...].
+
 
 Devices are added similarly, using `deviceManager` as the `kind`, like so:
 ```{literalinclude} ../../tests/unit_tests/valid_example_config/plans_and_devices.yaml

docs/how-to/authenticate.md → docs/how-to/1.2 authn-authz.md

@@ -2,16 +2,12 @@
 > If you are using `oauth2-proxy` to secure the Swagger UI documentation page, you can log out by visiting the `/logout` URL. For this to work correctly, ensure that the blueapi server is configured with 
 > `oidc.logout_redirect_endpoint` set to `/oauth2/sign_out`, which is required for `oauth2-proxy`.
 
-# Authenticate to BlueAPI-Cli
+# 1.2 Authenticate/Authorise to BlueAPI-Cli
 
-## Introduction
+## Authentication
 BlueAPI provides a secure and efficient way to interact with its services. This guide walks you through the steps to log in and log out using BlueAPI with OpenID Connect (OIDC) authentication.
 
-## Configuration
-
-:::{seealso}
-[Configure the Application](./configure-app.md)
-:::
+### Configuration
 
 Here is an example configuration for authenticating to p46-blueapi:
 ```{literalinclude} ../../tests/unit_tests/valid_example_config/client_auth.yaml
@@ -22,7 +18,7 @@ Here is an example configuration for authenticating to p46-blueapi:
 
 ---
 
-## Log In
+### Log In
 
 1. Execute the login command:
 
@@ -43,7 +39,7 @@ Here is an example configuration for authenticating to p46-blueapi:
 
 ---
 
-## Log Out
+### Log Out
 
 To log out and securely remove the cached access token, follow these steps:
 
@@ -63,3 +59,16 @@ To log out and securely remove the cached access token, follow these steps:
    ```
    Logged out
    ```
+
+## Authorization 
+
+The authorization provider is opa.diamond.ac.uk
+
+### Configuration 
+
+An example configuration to to customise authorization is shown in this `config.yaml` file. 
+
+```{literalinclude} ../../tests/system_tests/config.yaml
+:language: yaml
+```
+Else, the default values in `OpaConfig` class defined in `config.py` will be used. 

docs/how-to/configure-logging.md → docs/how-to/1.3 configure-logging.md

@@ -1,5 +1,5 @@
 
-# Configure Logging
+# 1.3 Configure Logging
 
 By default BlueAPI will log to stdout at the [INFO level](https://docs.python.org/3/library/logging.html#logging-levels), but can be reconfigured to log at any level, and to output to Graylog.
 
@@ -8,7 +8,7 @@ When logging to [Graylog](https://graylog.org) is enabled, BlueAPI will also con
 ## BlueAPI Cofiguration
 
 :::{seealso}
-[Configure the Application](./configure-app.md)
+[Configure the Application](./1.%20configure-app.md)
 :::
 
 An example logging config is shown below:

docs/how-to/integrate-with-numtracker.md → docs/how-to/1.4 integrate-with-numtracker.md

@@ -1,12 +1,12 @@
-# Integrate With Numtracker
+# 1.4 Integrate With Numtracker
 
 [Numtracker](https://github.com/DiamondLightSource/numtracker.git) is a DLS API that can be used to coordinate where detectors write their data.
 
 ## Prequisities
 
 You will need to [get numtracker itself configured for your instrument](https://github.com/DiamondLightSource/numtracker/wiki/new_beamline). 
 
-Blueapi needs [valid authentication configured](./authenticate.md) to communicate with numtracker. It will propogate its auth token so both blueapi and numtracker should be aware of who the user is and that they have permission to be on the instrument sessions (visits) that are intended for use.
+Blueapi needs [valid authentication configured](../how-to/1.2%20authn-authz.md) to communicate with numtracker. It will propogate its auth token so blueapi is aware of who the user is and that they have permission to be on the instrument sessions (visits) that are intended for use. Numtracker will not be aware of the user. 
 
 ## Configuration
 

docs/how-to/edit-live.md → docs/how-to/1.5 edit-live.md

@@ -1,4 +1,4 @@
-# Edit Plans and Device Live
+# 1.5 Edit Plans and Device Live
 
 You may want to tweak/edit your plans and devices live, i.e. without having to make a new release of a Python module, `pip install` it and restart blueapi. Blueapi can be configured to use a special directory called the "scratch area" where source code can be checked out and installed in [development mode](https://setuptools.pypa.io/en/latest/userguide/development_mode.html).
 

docs/how-to/1.6 configure-tiled.md

@@ -0,0 +1,28 @@
+# 1.6 Tiled
+
+Tiled is a server and data management system used to store and organise experimental data through a web API. It is run as a (mostly) centralised service. The RunEngine inserts bluesky documents directly into Tiled. At the moment, NeXus files are written from the messages published on the message bus. Tiled should contain the same information that is in the NeXus files but in due course, Tiled should be the one source of truth. 
+
+## Configuration
+
+An example configuration to enable tiled is shown in this `config.yaml` file. 
+
+```{literalinclude} ../../tests/system_tests/config.yaml
+:language: yaml
+```
+
+The options to configure Tiled are in the `TiledConfig` model which can be found in the [config.py file](../../src/blueapi/config.py). Apart from the url, the main aspect is authentication. This is either an ApiKey:
+
+```
+authentication: secret-key 
+```
+
+or service account details:
+
+```
+authentication:
+    client_id: "client-id"
+    client_secret: "secret"
+    token_url: "url"
+```
+
+The values in the `config.yaml` are the default ones use if the user doesn't provide them. 

docs/how-to/deploy-new-blueapi-version.md

@@ -0,0 +1,8 @@
+# Deploying a new version of BlueAPI
+
+Follow these [instructions](https://gitlab.diamond.ac.uk/sscc-docs/developer-guide/-/blob/4f28b0c4b17e34ae19faf698024bceb6e20992c9/topics/kubernetes/docs/tutorials/local_helm_on_argocd.md) for changing/testing new Blueapi deployments. 
+
+
+Links to the relevant services and deployment repositories:
+- Testing (pxx) beamlines: [epics-containers](https://github.com/epics-containers)
+- Other beamlines: [GitLab folder](https://gitlab.diamond.ac.uk/controls/containers/beamline)

docs/how-to/just-run-blueapi-and-services-locally.md

@@ -0,0 +1,37 @@
+# Run BlueAPI and connect to services locally
+
+For development purposes, it can be useful to run BlueAPI and adjacent services (Numtracker, Tiled, OPA etc.) locally, i.e. not in a devcontainer. Following the steps in this page will allow you to launch an instance of NumTracker, RabbitMQ, Keycloak, Tiled, OPA and a number of IOCs, in detached mode. This can be useful for learning about the stack, running system tests checking if changes during development propagate as expected etc. 
+
+Before starting, ensure you have followed the [Installation instructions](../tutorials/1.%20installation.md).
+
+1. Before starting, run: `module load uv just docker-compose/5.4.0`  in the terminal. This will ensure you have the required packages.
+
+2. The default command `just` into the terminal will do the following:
+    - A. initialise the example-services repo
+    - B. launch an instance of NumTracker, RabbitMQ, Keycloak, Tiled, OPA and a number of IOCs in detached mode
+    - C. set required EPICS environmental variables
+    - D. start the BlueAPI server using the the config in `tests/system_tests/config.yaml`
+
+3. To run the above separately use the following commands:
+    - 2A and 2B: `just compose`
+    - 2C and 2D: `just serve`
+
+4. In a new terminal window, to run unit and system tests respectively: `just unit` and `just system`
+
+5. Other commands available (use `just --list` to see them)
+    - `just run PLAN PARAMS`: provide session, plan and parameter details to run a plan
+    - `just lint`: will run all the precommit checks, update the BlueAPI schema, run pyright
+    - `just coverage`: generate code coverage report
+    - `just repl`: will give you a repl with pre-configured and logged in client
+    - `just compose down`: tear down adjacent services
+
+Channel Access
+
+To log in through the BlueAPI CLI:
+
+1. Run `blueapi login` (if you want to run a plan with stomp config, add the `-c tests/system_tests/config.yaml` parameter)
+2. Follow the login prompted to Keycloak, then log in with the username `admin` and password `admin`
+3. When prompted by Keycloak, grant BlueAPI access to the listed privileges
+4. Run `blueapi controller plans` to check that the log in has succeeded
+
+By default the BlueAPI instance will be available via the OAuth2 proxy at `localhost:4180`, and Tiled through its OAuth2 proxy at `localhost:4181`.

docs/how-to/run-container.md

@@ -19,3 +19,12 @@ podman run ghcr.io/diamondlightsource/blueapi:latest
 ```
 
 To get a released version, use a numbered release instead of `latest`.
+
+## Starting the message bus
+
+The worker requires a running instance of the message bus, which in this case is RabbitMQ. The easiest way to start it is
+ to `compose` the services in `tests/system_tests/compose.yaml`
+
+```sh
+docker compose -f tests/system_tests/compose.yaml run rabbitmq --detach
+```

docs/tutorials/run-with-dev.md → docs/how-to/run-with-dev.md

@@ -10,7 +10,7 @@ Ensure you are inside your virtual environment:
 source venv/bin/activate
 ```
 
-You will need to follow the instructions for setting up RabbitMQ as in [instructions for setting up RabbitMQ](run-bus.md).
+You will need to follow the instructions for setting up RabbitMQ as in [instructions for setting up RabbitMQ](../tutorials/5.%20config-add-rabbitmq-and-run-plan-cli.md).
 
 The worker will be available from the command line (`blueapi serve`), but can be started from vscode with additional
 debugging capabilities.
@@ -22,5 +22,5 @@ debugging capabilities.
 [debug in vscode](../images/debug-vscode.png)
 
 :::{seealso}
-[Scratch Area](../how-to/edit-live.md) for in-the-loop development of plans and devices
+[Scratch Area](../how-to/1.5%20edit-live.md) for in-the-loop development of plans and devices
 :::

docs/how-to/use-persistent-volumes.md

@@ -5,7 +5,7 @@ BlueAPI can use persistent volumes as a scratch area. This allows the user to re
 ## Configuration
 
 :::{seealso}
-[Configure the Application](./configure-app.md)
+[Configure the Application](../how-to/1.%20configure-app.md)
 :::
 
 The relevant configuration is below:

docs/how-to/write-devices.md

@@ -23,5 +23,5 @@ The return type annotation `-> MyTypeOfDetector` is required as blueapi uses it
 The device is created via a function rather than a global to preserve side-effect-free imports. Each device must have its own factory function.
 
 :::{seealso}
-[Numtracker integration](./integrate-with-numtracker.md) for how to configure where detectors write files.
+[Numtracker integration](../how-to/1.4%20integrate-with-numtracker.md) for how to configure where detectors write files.
 :::

docs/reference/dev-commands.md

@@ -0,0 +1,14 @@
+# Commands for dev tooling
+
+This page outlines useful commands for dev tooling. More information can be found in the [python-copier-template docs](https://github.com/DiamondLightSource/python-copier-template/tree/main/docs). 
+
+The table below outlines useful commands with instructions for download on Diamond workstations.
+
+| Command           | Installation  | Example of use|
+| -------------     | ------------- | ------------- |
+| `podman`          | [Dev-Guide link](https://dev-guide.diamond.ac.uk/containers/tutorials/podman/)  | `podman run ghcr.io/diamondlightsource/blueapi:latest` to pull the blueapi container from the GitHub container registry |
+| `docker-compose`  | [Dev-Guide link](https://dev-guide.diamond.ac.uk/epics-containers/reference/setup/#docker-compose)  | `docker compose -f tests/system_tests/compose.yaml up -d` to spin up dummy versions of blueapi associated services |
+| `uv`              | `module load uv`  | `uv run --with blueapi path/to/script.py` to run a standalone script |
+| `prek`            | Run `uv run prek install` the first time the repo is cloned to set up pre-commit checks  | `uv run prek` or `uv run prek --all-files`  |
+| `tox`             |  `uv tool install tox` | `tox -e tests -- tests/unit_tests/test_config.py::test_config_yaml_parsed` is an example of running a specific test and `tox -e tests` will run all unit tests|
+| `just`            | `uv tool rust-just`  | `just lint` will run all pre-commit checks|

docs/resources/enabled_plugins

@@ -0,0 +1 @@
+[rabbitmq_stomp].

docs/tutorials/installation.md → docs/tutorials/1. installation.md

@@ -1,4 +1,4 @@
-# Installation
+# 1. Installation
 
 ## Check your version of python
 
@@ -8,6 +8,10 @@ typing into a terminal:
 python3 --version
 ```
 
+## Clone the blueapi repository locally 
+
+Clone the blueapi repository to your local scratch directory and `cd` into it.
+
 ## Create a virtual environment
 
 It is recommended that you install into a “virtual environment” so this
@@ -35,3 +39,11 @@ You can check the version that has been installed by typing:
 ``` 
 blueapi --version
 ```
+
+Troubleshooting:
+- If you encounter a keycloak hook 137 error, this is because of there not being enough resources. You may need to close things that are running in the background.
+- You may encounter 
+```
+Error response from daemon: rootlessport listen tcp 0.0.0.0:1883: bind: address already in use
+```
+This can arise from `activemq-for-dummy` which is used across DLS for GDA development. You will need to stop processes that have conflicting ports.

docs/tutorials/2. quickstart.md

@@ -0,0 +1,39 @@
+# 2. Quickstart guide
+
+Blueapi acts as a worker that can run Bluesky plans against devices for a specific laboratory setup. It can control devices to collect data and export events to tell downstream services about the data it has collected.
+
+You should be in the blueapi directory that you have cloned. From here, if you open VSCode, you will probably be prompted to 'Reopen in Container' which you can do and then run the following commands to start the worker. 
+
+If you not, you can still run the following from the venv created in page 1 but you may need to run the command below to have the dependencies required for this tutorial.
+
+```
+pip install .[demo]
+```
+
+## Start Worker
+
+To start the worker:
+```
+blueapi serve
+```
+This will use all the defaults in `src/blueapi/config.py`
+
+## Basic Introspection
+
+Blueapi comes with a minimal CLI client for basic control/debugging. We can use this to see which plans and devices are available. In a new terminal window, run the following commands. 
+
+If working in a venv, you may need to reactivate it. To do this, assuming you are still in the blueapi repo in your terminal, use:
+```
+source .venv/bin/activate
+```
+
+Now, run the following to see what plans and devices are available:
+
+``` 
+blueapi controller plans
+blueapi controller devices
+```
+
+We should now have an empty context.
+
+See also [Full CLI reference](../reference/cli.md)