Knowledge Base: Tabular Data, Metadata, and Query Filtering

We are excited to announce a powerful new feature in Voiceflow's Knowledge Base: the ability to handle tabular data, enrich all data sources with metadata, and perform advanced query filtering. This enhancement provides our users with more flexibility, precision, and control over their AI agents, ensuring they deliver the most relevant and contextually accurate responses.

Key Features

Tabular Data Support
With the introduction of tabular data support, users can now upload structured data in table format. This is ideal for scenarios where information is best represented in rows and columns, such as product catalogs, inventories, or detailed records. This structured format allows for more organized and efficient data management within your Voiceflow agents.

Metadata Enrichment
Metadata is a critical component for enhancing the context and relevance of your data. You can now attach key-value pairs to all your data sources, providing additional layers of information that can be used to fine-tune your AI agent’s responses. Whether it’s categorizing products, tagging items with specific attributes, or adding detailed descriptions, metadata makes your data more insightful and searchable.

Advanced Query Filtering
Our new query filtering capabilities enable users to perform precise searches within their data using a robust set of operators. You can apply various filters to your data, such as:

Equality and Comparison: $eq, $ne, $gt, $gte, $lt, $lte
Array Operations: $in, $nin, $all
Logical Operators: $and, $or

This means you can now create complex queries to retrieve exactly what you need, whether it’s finding products under a certain price, filtering by multiple tags, or combining multiple conditions.

How to Use

Inserting Metadata to Your Data Sources

Voiceflow supports adding metadata to your data sources through three methods: FILE upload, URL upload, and TABLE upload. Depending on the type of document being uploaded, follow the respective method below:

FILE Upload

curl --request POST \ 
--url 'https://api.voiceflow.com/v1/knowledge-base/docs/upload?overwrite=true' \ 
--header 'Authorization: YOUR_DM_API_KEY' \
--header 'Content-Type: multipart/form-data' \ 
--header 'User-Agent: insomnia/8.0.0' \  
--form 'file=@/path/to/your/file.pdf' \  
--form 'metadata={"inner": {"text": "some test value", "price": 5, "tags": ["t1", "t2", "t3", "t4"]}}'

URL Upload

curl --request POST  
  --url '<https://api.voiceflow.com/v1/knowledge-base/docs/upload>  
  --header 'Authorization: YOUR_DM_API_KEY'  
  --header 'Content-Type: application/json'  
  --data '{  
    "data": {  
        "type": "url",  
        "url": "<https://example.com/">,  
        "metadata": {"test": 5}  
    }  
}'

TABLE Upload

curl --request POST \
  --url 'https://api.voiceflow.com/v1/knowledge-base/docs/upload/table' \
  --header 'Authorization: YOUR_DM_API_KEY' \
  --header 'Content-Type: application/json' \
  --data '{
    "data": {
        "name": "products",
        "schema": {
            "searchableFields": ["name", "description"],
            "metadataFields": ["developer", "project"]
        },
        "items": [
            {
                "name": "example_name",
                "description": "example_description",
                "developer": {
                    "name": "Jane Doe",
                    "level": "senior",
                    "skills": ["Python", "JavaScript"],
                    "languages": [
                        {
                            "name": "Russian"
                        },
                        {
                            "name": "German"
                        }
                    ]
                },
                "project": {
                    "name": "AI Development",
                    "deadline": "2024-12-31"
                }
            }
        ]
    }
}'

Practical Query Examples
Here are some practical examples to demonstrate the power of query filtering using the Query API:

Match All Specific Tags: Identify chunks that include every specified tag in the list.

{  
    "filters": {  
        "inner.tags": {  
            "$all": ["t1", "t2"]  
        }  
    }  
}

Match Any of the Specified Tags: Find chunks containing any of the tags listed.

{  
    "filters": {  
        "inner.tags": {  
            "$in": ["t1", "t2"]  
        }  
    }  
}

Exclude Chunks With Certain Tags: Filter out chunks that include any of the tags specified.

{
    "filters": {
        "developer.tags": {
            "$nin": ["t1", "t2"]
        }
    }
}

Combination of Conditions: Search for chunks that either contain a specific tag or match a text value exactly.

{
    "filters": {
        "$or": [
            {
                "developer.tags": {
                    "$in": ["t1"]
                }
            },
            {
                "developer.name": {
                    "$eq": "Jane Doe"
                }
            }
        ]
    }
}

Important Update

We have also updated the route for document upload from https://api.voiceflow.com/v3alpha/knowledge-base/docs/upload to https://api.voiceflow.com/v1/knowledge-base/docs/upload. The v3alpha version is still operational, but we recommend moving over to the new URL for improved performance and support.
Stay tuned for more updates and happy agent building!