feat: add guide for later usecases
Browse files- guide.md +80 -0
- metadata.jsonl +0 -0
- test.ipynb +0 -0
guide.md
ADDED
|
@@ -0,0 +1,80 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Gaia Agent Evaluation Guide
|
| 2 |
+
|
| 3 |
+
This guide will walk you through the setup process for running the sample code and evaluating your agent using Gaia results.
|
| 4 |
+
|
| 5 |
+
## Step 1: Configure API Keys
|
| 6 |
+
|
| 7 |
+
Before anything else, make sure you configure your secret keys in the **Space Settings** section.
|
| 8 |
+
|
| 9 |
+
- Log into each required platform.
|
| 10 |
+
- Locate and input your API keys in the designated fields.
|
| 11 |
+
|
| 12 |
+
## Step 2: Set Up Supabase
|
| 13 |
+
|
| 14 |
+
1. **Log in to Supabase**.
|
| 15 |
+
2. Navigate to your **space**, then go to your **project**.
|
| 16 |
+
3. Open the **SQL Editor**, paste the SQL code below, and run it to create the necessary table and function.
|
| 17 |
+
|
| 18 |
+
### 📦 SQL Code – Creating Tables and Functions
|
| 19 |
+
|
| 20 |
+
```sql
|
| 21 |
+
-- Enable pgvector if not already enabled
|
| 22 |
+
create extension if not exists vector;
|
| 23 |
+
|
| 24 |
+
-- Create the documents table (if not already done)
|
| 25 |
+
create table if not exists documents (
|
| 26 |
+
id bigserial primary key,
|
| 27 |
+
content text,
|
| 28 |
+
metadata jsonb,
|
| 29 |
+
embedding vector(768) -- Make sure this matches your model's embedding dimension
|
| 30 |
+
);
|
| 31 |
+
|
| 32 |
+
-- Create the match_documents function
|
| 33 |
+
create or replace function match_documents (
|
| 34 |
+
query_embedding vector(768),
|
| 35 |
+
match_count int default 5,
|
| 36 |
+
filter jsonb default '{}'
|
| 37 |
+
)
|
| 38 |
+
returns table (
|
| 39 |
+
id bigint,
|
| 40 |
+
content text,
|
| 41 |
+
metadata jsonb,
|
| 42 |
+
similarity float
|
| 43 |
+
)
|
| 44 |
+
language plpgsql
|
| 45 |
+
as $$
|
| 46 |
+
begin
|
| 47 |
+
return query
|
| 48 |
+
select
|
| 49 |
+
id,
|
| 50 |
+
content,
|
| 51 |
+
metadata,
|
| 52 |
+
1 - (embedding <=> query_embedding) as similarity
|
| 53 |
+
from documents
|
| 54 |
+
where metadata @> filter
|
| 55 |
+
order by embedding <=> query_embedding
|
| 56 |
+
limit match_count;
|
| 57 |
+
end;
|
| 58 |
+
$$;
|
| 59 |
+
```
|
| 60 |
+
4. After running the above, execute this command to ensure Supabase’s API layer (PostgREST) refreshes its internal schema cache:
|
| 61 |
+
```sql
|
| 62 |
+
NOTIFY pgrst, 'reload config';
|
| 63 |
+
```
|
| 64 |
+
## Step 3: Populate the Database
|
| 65 |
+
|
| 66 |
+
To enable document retrieval, you need to populate the database with example entries:
|
| 67 |
+
|
| 68 |
+
- Open and run the **test.ipynb** Jupyter notebook.
|
| 69 |
+
|
| 70 |
+
- This script reads from the **metadata.jsonl** file and inserts the examples into the documents table.
|
| 71 |
+
|
| 72 |
+
- This adds a Basic Retrieval capability to your agent, enhancing its performance.
|
| 73 |
+
|
| 74 |
+
## Step 4: Run the Evaluation
|
| 75 |
+
|
| 76 |
+
Once the database is set up and filled with data:
|
| 77 |
+
|
| 78 |
+
- Proceed to the Evaluation section in your project.
|
| 79 |
+
|
| 80 |
+
- Run the evaluation script to test and score your agent’s performance.
|
metadata.jsonl
ADDED
|
The diff for this file is too large to render.
See raw diff
|
|
|
test.ipynb
ADDED
|
The diff for this file is too large to render.
See raw diff
|
|
|