Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/PULL_REQUEST_TEMPLATE.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

--- PR TEMPLATE INSTRUCTIONS (1) ---

Looking to submit a Apache Hamilton Dataflow to the apache-hamilton-contrib module? If so go the the `Preview` tab and select the appropriate sub-template:
Looking to submit a Apache Hamilton Dataflow to the apache-hamilton-contrib module? If so go to the `Preview` tab and select the appropriate sub-template:
* [apache-hamilton-contrib template](?expand=1&template=HAMILTON_CONTRIB_PR_TEMPLATE.md)

Else, if not, please remove this block of text.
Expand Down
4 changes: 2 additions & 2 deletions contrib/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@ ready to be used in your own projects. They are user-contributed and maintained,
the goal of making it easier for you to get started with Apache Hamilton.

We expect this collection to grow over time, so check back often! As dataflows become mature we
will move them into the official sub-package of the respository and become maintained by the
will move them into the official sub-package of the repository and become maintained by the
Apache Hamilton team.

### Usage
Expand Down Expand Up @@ -141,7 +141,7 @@ Standards:

To get started with development, you'll want to first fork the hamilton repository from the github UI.

Then, clone it locally and install the package in editable mode, ensuring you install any dependencies required for the initilization script
Then, clone it locally and install the package in editable mode, ensuring you install any dependencies required for the initialization script
```bash
cd hamilton # Your fork
pip install -e "./contrib[contribute]" # Note that this package lives under the `contrib` folder
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,7 @@ This also accepts the following overrides:

Note that this has two modes:

1. (`include_embeddings=False`, or not specified) Just quries the caption for the image. You can request `generated_caption`, which will give embeddings for the caption.
1. (`include_embeddings=False`, or not specified) Just queries the caption for the image. You can request `generated_caption`, which will give embeddings for the caption.
2. (`include_embeddings=True`): also gets the caption for the image, and the embeddings the caption. This requires `caption_embeddings` to be one of the variables called.

If you want all the metadata about this run wrapped up in a dictionary (say, to save externally), you can request `metadata` which will return a dict with the following keys:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -45,7 +45,7 @@ This module doesn't receive configurations.
### Inputs
- `query`: The query for which you want code generated.
- `api_key`: Set the OpenAI API key to use. If None, read the environment variable `OPENAI_API_KEY`
- `code_language`: Set the code language to generate the reponse in. Defaults to `python`
- `code_language`: Set the code language to generate the response in. Defaults to `python`

### Overrides
- `prompt_template_to_generate_code`: Create a new prompt template with the fields `query` and `code_language`.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,7 @@ This module doesn't receive configurations.

## Overrides
- `base_models`: Create a list of sklearn-compatible models to fit and evaluate ([docs](https://nixtla.github.io/statsforecast/src/core/models.html))
- `lag_transforms` and `date_features`: Should be overriden to pass values relevant to your dataset
- `lag_transforms` and `date_features`: Should be overridden to pass values relevant to your dataset
- `evaluation_metrics`: Set the list of Nixtla-compatible metrics to use during cross-validation ([examples](https://github.com/Nixtla/utilsforecast/blob/main/utilsforecast/losses.py))


Expand Down
2 changes: 1 addition & 1 deletion docs/how-tos/pre-commit-hooks.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ This page gives an introduction to pre-commit hooks and how to use custom hooks
## What are pre-commit hooks?
A pre-commit hook is a script or command that's executed automatically before making a commit. The goal of these hooks is to standardize code formatting and catch erroneous code before being committed. For example, popular hooks include ensuring files have no syntax errors, sorting imports, and normalizing line breaks.

Note that it's different from testing, which focuses on the behavior of the code. You can think of pre-commit hooks as checks and formatting you would do everytime you save a file.
Note that it's different from testing, which focuses on the behavior of the code. You can think of pre-commit hooks as checks and formatting you would do every time you save a file.

## Add pre-commit hooks to your project
Hooks are a mechanism of the `git` version control system. You can find your project's hooks under the `.git/hooks` directory (it might be hidden by default). There should be many files with the `.sample` extension that serve as example scripts.
Expand Down
2 changes: 1 addition & 1 deletion docs/how-tos/use-in-jupyter-notebook.md
Original file line number Diff line number Diff line change
Expand Up @@ -108,7 +108,7 @@ def log_avg_3wk_spend(avg_3wk_spend: pd.Series) -> pd.Series:
We then have to create a "temporary python module" to house it in. We do this by importing `ad_hoc_utils` and then calling the `create_temporary_module` function, passing in the functions we want, and providing a name for the module we're creating.

```python
# Step 2 - create a temporary modeul to house all notebook functions
# Step 2 - create a temporary module to house all notebook functions
from hamilton import ad_hoc_utils
temp_module = ad_hoc_utils.create_temporary_module(
log_avg_3wk_spend, module_name='function_example')
Expand Down
2 changes: 1 addition & 1 deletion docs/integrations/fastapi.md
Original file line number Diff line number Diff line change
Expand Up @@ -57,7 +57,7 @@ FastAPI already does a great job at automating API documentation by integrating
## Apache Hamilton + FastAPI
Adding Apache Hamilton to your FastAPI server can provide a better separation between the dataflow and the API endpoints. Each endpoint can use `Driver.execute()` to request variables and wrap results into an HTTP response. Then, data transformations and interactions with resources (e.g., database, web service) are curated into standalone Python modules and decoupled from the server code.

Since Apache Hamilton dataflows will run the same way inside or outside FastAPI, you can write simpler unit tests for Hamilton functions without defining a mock server and client. Additionnally, visualizations for the defined Apache Hamilton dataflows can be added to the FastAPI [Swagger UI documentation](https://fastapi.tiangolo.com/features/#automatic-docs). They will remain in sync with the API behavior because they are generated from the code.
Since Apache Hamilton dataflows will run the same way inside or outside FastAPI, you can write simpler unit tests for Hamilton functions without defining a mock server and client. Additionally, visualizations for the defined Apache Hamilton dataflows can be added to the FastAPI [Swagger UI documentation](https://fastapi.tiangolo.com/features/#automatic-docs). They will remain in sync with the API behavior because they are generated from the code.

### Example
In this example, we'll build a backend for a PDF summarizer application.
Expand Down
2 changes: 1 addition & 1 deletion examples/LLM_Workflows/NER_Example/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@ this can be used when trying to find the most relevant text to pass to an LLM in
context.

The pipeline we create can be seen in the image below.
![pipeine](ner_extraction_pipeline.png)
![pipeline](ner_extraction_pipeline.png)

To run this in a notebook:

Expand Down
2 changes: 1 addition & 1 deletion examples/LLM_Workflows/langchain_comparison/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,7 @@ Files prefixed with `hamilton_` are the Apache Hamilton equivalent of the exampl
As you browse the files you'll see that:

1. LangChain's focus is on hiding details and making code terse.
2. Apache Hamilton's focus instead is on making code more readable, maintainable, and importantly customizeable.
2. Apache Hamilton's focus instead is on making code more readable, maintainable, and importantly customizable.

## Implications
Don't be surprised that Apache Hamilton's code is "longer" - that's by design. There is
Expand Down
2 changes: 1 addition & 1 deletion examples/LLM_Workflows/modular_llm_stack/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -72,7 +72,7 @@ To change embedding service, you can use the following:

To change vector database you need to pass a JSON config argument:
- LanceDB: `--vector_db=lancedb --vector_db_config='{"uri": "data/lancedb"}'`
- Weaviate: `--vector_db=weaviate --vector_db_config='{"url": "http://locahost:8080/"}'`
- Weaviate: `--vector_db=weaviate --vector_db_config='{"url": "http://localhost:8080/"}'`
- Pinecone: `--vector_db=pinecone --vector_db_config='{"environment": "ENVIRONMENT", "api_key": "API_KEY"}'`
- Marqo: `--vector_db=marqo --vector_db_config='{"url":"http://localhost:8882"}' --other_input_kwargs '{"index_name":"hamilton"}'`
- Qdrant: `--vector_db=qdrant --vector_db_config='{"url":"http://localhost:6333"}'`
Expand Down
2 changes: 1 addition & 1 deletion examples/LLM_Workflows/pdf_summarizer/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ under the License.
-->

# (Yet another) LLM PDF Summarizer 📝
Here's an extensible and production-ready PDF summarizer that you can run anywhere! The frontend uses streamlit, which communicates with a FastAPI backend powered by Apache Hamilton. You give it a PDF file via the browser app and it returns you a text summary using the OpenAI API. If you want, you skip the browser inteface and directly access the `/summarize` endpoint with your document! Everything is containerized using Docker, so you should be able to run it where you please 🏃.
Here's an extensible and production-ready PDF summarizer that you can run anywhere! The frontend uses streamlit, which communicates with a FastAPI backend powered by Apache Hamilton. You give it a PDF file via the browser app and it returns you a text summary using the OpenAI API. If you want, you skip the browser interface and directly access the `/summarize` endpoint with your document! Everything is containerized using Docker, so you should be able to run it where you please 🏃.

## Why build this project?
This project shows how easy it is to productionize Apache Hamilton. Its function-centric declarative approach makes the code easy to read and extend. We invite you to clone the repo and customize to your needs! We are happy to help you via [Slack](https://hamilton-opensource.slack.com/join/shared_invite/zt-1bjs72asx-wcUTgH7q7QX1igiQ5bbdcg) and are excited to see what you build 😁
Expand Down
2 changes: 1 addition & 1 deletion examples/async/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -70,7 +70,7 @@ pip install apache-hamilton[ui]
hamilton ui
```

This assumes a project (1) exists -- if you want a different one you can go the the UI and create one and/or set it in the code.
This assumes a project (1) exists -- if you want a different one you can go to the UI and create one and/or set it in the code.


## How it works
Expand Down
2 changes: 1 addition & 1 deletion examples/cli/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,7 @@ Test the installation with
- `validate`: calls `Driver.validate_execution()` for a set of `inputs` and `overrides` passed through the `--context` option.
- `view`: calls `dr.display_all_functions()` on the built `Driver`
- `version`: generates node hashes based on their source code, and a dataflow hash from the collection of node hashes.
- `diff`: get a diff of added/deleted/edited nodes between the current version of Python modules and another git reference (`default=HEAD`, i.e., the last commited version). You can get a visualization of the diffs
- `diff`: get a diff of added/deleted/edited nodes between the current version of Python modules and another git reference (`default=HEAD`, i.e., the last committed version). You can get a visualization of the diffs

## Options
- all commands receive `MODULES` which is a list of path to Python modules to assembled as a single dataflow
Expand Down
2 changes: 1 addition & 1 deletion examples/dagster/dagster_code/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@ This project is adapted from the official [Dagster tutorial](https://docs.dagste
- `pyproject.toml` and `setup.py` define how to install the `tutorial/` Dagster project.
- `tutorial/assets.py` defines the data assets to compute and materialize.
- `tutorial/__init__.py` register the data assets, jobs, and resources for the orchestrator.
- `tutorial/resources/` contains informations to connect to external resources and API.
- `tutorial/resources/` contains information to connect to external resources and API.

## Instructions
1. Install the Dagster project as a Python module
Expand Down
2 changes: 1 addition & 1 deletion examples/dagster/hamilton_code/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ The Apache Hamilton refactor achieves the same data transformations. The project
- `run.py` contains the "driver code" to load and execute the dataflow (equivalent to Dagster `tutorial/__init__.py`).
- `notebook.ipynb` contains code to execute the dataflow (similar code to `run.py`), but allows for an interactive exploration of results.

The directory also include `mock_api.py` which simulates a `ressource` in the Dagster tutorial, and `dataflow.png` which is an artifact generated by the Apache Hamilton `Driver`.
The directory also include `mock_api.py` which simulates a `resource` in the Dagster tutorial, and `dataflow.png` which is an artifact generated by the Apache Hamilton `Driver`.

## Instructions
1. Run the code
Expand Down
2 changes: 1 addition & 1 deletion examples/dlt/slack/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -95,7 +95,7 @@ source = slack_source()
source.access_logs.selected = True
```

The conversations resource can be used to load the list of available conversations and will allways bring all
The conversations resource can be used to load the list of available conversations and will always bring all
the available conversations.

The history resource can be used to load the history of a conversation. By default all the conversations will be
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -64,7 +64,7 @@ Not having lineage or visibility into what data is being used for what purpose c


## Deployment: Offline vs online vs streaming needs
This is a big topic. We wont do it justice here, but let's try to give a brief overview of two main problems:
This is a big topic. We won't do it justice here, but let's try to give a brief overview of two main problems:

(1) There are a lot of different deployment needs when you get something to production. For example, you might want to:
- run a batch job to generate features for a model
Expand Down Expand Up @@ -151,7 +151,7 @@ Jump into each directory and read the README, it'll explain how the example is s
# What are extensions/uses not shown here but we know you can do them
Here are two ideas that come to mind:

1. Streaming settings. Given the examples, it should be clear how to make it possbile to use Apache Hamilton in a streaming setting.
1. Streaming settings. Given the examples, it should be clear how to make it possible to use Apache Hamilton in a streaming setting.
2. How to ask Apache Hamilton what features are needed as input to know what to request from the feature store. With tags, and
querying the DAG at the start of the app, you could dynamically ask Apache Hamilton what's required and then only go to the
feature store for that data. If this type of example would be of interest, let us know.
Original file line number Diff line number Diff line change
Expand Up @@ -55,10 +55,10 @@ dataquality checks using Pandera.

Important not, there are two aggregations features defined: `age_mean` and `age_std_dev`, that are computed on the
`age` column. These make sense to compute in an offline setting as you have all the data, but in an online setting where
you'd be performing inference, that doesn't makse sense. So for the online case, these computations be "overridden" in
you'd be performing inference, that doesn't make sense. So for the online case, these computations be "overridden" in
`fastapi_server.py` with the values that were computed in the offline setting that you have stored (as mentioned above
and below it's up to you how to store them/sync them). The nice thing in Apache Hamilton is that we can also "tag" these two
feature transforms with information to indicate to someone reading the code, that they should be overriden in the
feature transforms with information to indicate to someone reading the code, that they should be overridden in the
online feature computation context.

## etl.py
Expand All @@ -75,7 +75,7 @@ Note, you need to store `age_mean` and

## named_model_feature_sets.py
Rather than hardcoding what features the model should have in two places, we define
it in a single place and import it where needed; this is simple if you can share the code eaisly.
it in a single place and import it where needed; this is simple if you can share the code easily.
However, this is something you'll have to determine how to best do in your set up. There are many ways to do this,
come ask in the [slack channel](https://join.slack.com/t/hamilton-opensource/shared_invite/zt-2niepkra8-DGKGf_tTYhXuJWBTXtIs4g)
if you need help.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -49,7 +49,7 @@ dataquality checks using Pandera.

Important not, there are two aggregations features defined: `age_mean` and `age_std_dev`, that are computed on the
`age` column. These make sense to compute in an offline setting as you have all the data, but in an online setting where
you'd be performing inference, that doesn't makse sense. So for the online case, in this example, they are "omitted" from
you'd be performing inference, that doesn't make sense. So for the online case, in this example, they are "omitted" from
being nodes in the online context via `@config.when_not_in`. Instead, they are populated via the `online_loader.py` which
goes to the feature store for them. `fastapi_server.py` then is a little simpler than in Scenario (1), as it just
delegates all work to Apache Hamilton for getting features.
Expand All @@ -72,7 +72,7 @@ Note, `age_mean` and `age_std_dev` are pushed to the feature store in this examp

## named_model_feature_sets.py
Rather than hardcoding what features the model should have in two places, we define
it in a single place and import it where needed; this is simple if you can share the code eaisly.
it in a single place and import it where needed; this is simple if you can share the code easily.
However, this is something you'll have to determine how to best do in your set up. There are many ways to do this,
come ask in the [slack channel](https://join.slack.com/t/hamilton-opensource/shared_invite/zt-2niepkra8-DGKGf_tTYhXuJWBTXtIs4g)
if you need help.
Expand Down
2 changes: 1 addition & 1 deletion examples/hamilton-tutorials/mpg-translation/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,7 @@ To get set up:
2. Go into the `examples/hamilton-tutorials/mpg-translation` directory.
2. Install Apache Hamilton. `pip install - requirements.txt`
3. Write code.
4. Excute via notebook or command line.
4. Execute via notebook or command line.

Or run it in Google Colab:

Expand Down
2 changes: 1 addition & 1 deletion examples/kedro/hamilton-code/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@ under the License.
The Apache Hamilton refactor is composed of a few files:
- `data_processing.py` and `data_science.py` contains regular Python functions to define the Apache Hamilton dataflow. This is equivalent to Kedro's `pipeline.py` **and** `nodes.py` files.
- `run.py` contains the "driver code" to load and execute the dataflow. There's no direct equivalent in the Kedro tutorial since it prefers using the CLI for execution.
- `noteboks/interactive.ipynb` contains the "driver code", similar to `run.py`, but uses [Apache Hamilton Jupyter Magics](https://hamilton.apache.org/how-tos/use-in-jupyter-notebook/#use-hamilton-jupyter-magic) to define the dataflow interactily in a notebook.
- `notebooks/interactive.ipynb` contains the "driver code", similar to `run.py`, but uses [Apache Hamilton Jupyter Magics](https://hamilton.apache.org/how-tos/use-in-jupyter-notebook/#use-hamilton-jupyter-magic) to define the dataflow interactively in a notebook.
- `tests/test_dataflow.py` includes tests equivalent to `tests/pipelines/data_science/test_pipeline.py` in the Kedro code.

## Instructions
Expand Down
2 changes: 1 addition & 1 deletion examples/parallelism/graceful_running/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ under the License.

This is a simple example of using the `GracefulErrorAdapter` in a parallelism example, where we might expect some component of an analysis to fail, but we'd still like to get as much data back as we can.

This example divides a large dataframe into smaller frames, and runs the same analysis on each of those frames. It then gathers the results at the end into a single frame. Any errors inside the paralellism block do not halt the total operation of the driver.
This example divides a large dataframe into smaller frames, and runs the same analysis on each of those frames. It then gathers the results at the end into a single frame. Any errors inside the parallelism block do not halt the total operation of the driver.

The user can define custom data splitting functions to process in the same sub-dag. In some ways, this is an example of how to do `@subdag` with `Parallelizable`.

Expand Down
2 changes: 1 addition & 1 deletion examples/ray/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -31,5 +31,5 @@ scaling Apache Hamilton on Ray, where `@ray.remote` is injected around
each Apache Hamilton function.

## creating tasks
For [this paralle task approach](https://hamilton.apache.org/concepts/parallel-task/) see [this
For [this parallel task approach](https://hamilton.apache.org/concepts/parallel-task/) see [this
example](https://github.com/apache/hamilton/tree/main/examples/LLM_Workflows/scraping_and_chunking) instead.
Loading