feat(helm): Add document chunking configuration
Add support for configurable document chunking parameters to Helm chart to match docker-compose and application capabilities. Changes: 1. values.yaml: - Add documentChunking section with chunkSize (512) and chunkOverlap (50) - Include comprehensive comments explaining chunking strategies - Positioned between vectorSync and qdrant sections 2. templates/deployment.yaml: - Add DOCUMENT_CHUNK_SIZE and DOCUMENT_CHUNK_OVERLAP env vars - Always set (not conditional), used by vector sync processor - Environment variables follow same pattern as config.py defaults 3. README.md: - Add documentChunking parameter table in Vector Search section - Document chunking strategies (small/medium/large chunks) - Explain overlap recommendations (10-20% of chunk size) Validation: - helm lint: Passes - helm template: Environment variables correctly generated - Custom values: Work as expected (tested with chunkSize=1024) - Always present: Not conditional on vectorSync.enabled This maintains feature parity between Helm and docker-compose deployments, allowing users to tune chunking for their embedding models and use cases. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
@@ -219,6 +219,19 @@ Enable semantic search capabilities by deploying a vector database (Qdrant) and
|
|||||||
| `vectorSync.processorWorkers` | Number of concurrent processor workers | `3` |
|
| `vectorSync.processorWorkers` | Number of concurrent processor workers | `3` |
|
||||||
| `vectorSync.queueMaxSize` | Maximum queue size for pending documents | `10000` |
|
| `vectorSync.queueMaxSize` | Maximum queue size for pending documents | `10000` |
|
||||||
|
|
||||||
|
**Document Chunking Configuration:**
|
||||||
|
|
||||||
|
| Parameter | Description | Default |
|
||||||
|
|-----------|-------------|---------|
|
||||||
|
| `documentChunking.chunkSize` | Number of words per chunk for embedding | `512` |
|
||||||
|
| `documentChunking.chunkOverlap` | Number of overlapping words between chunks | `50` |
|
||||||
|
|
||||||
|
**Chunking Strategy:**
|
||||||
|
- **Small chunks (256-384)**: Better precision for searches, more storage overhead
|
||||||
|
- **Medium chunks (512-768)**: Balanced approach (recommended for most use cases)
|
||||||
|
- **Large chunks (1024+)**: Better context preservation, less precise matching
|
||||||
|
- **Overlap**: Should be 10-20% of chunk size to preserve context across boundaries
|
||||||
|
|
||||||
**Qdrant Vector Database:**
|
**Qdrant Vector Database:**
|
||||||
|
|
||||||
Qdrant is deployed as a subchart when `qdrant.enabled` is `true`. All configuration values are passed through to the [qdrant/qdrant](https://github.com/qdrant/qdrant-helm) chart.
|
Qdrant is deployed as a subchart when `qdrant.enabled` is `true`. All configuration values are passed through to the [qdrant/qdrant](https://github.com/qdrant/qdrant-helm) chart.
|
||||||
|
|||||||
@@ -158,6 +158,11 @@ spec:
|
|||||||
- name: VECTOR_SYNC_QUEUE_MAX_SIZE
|
- name: VECTOR_SYNC_QUEUE_MAX_SIZE
|
||||||
value: {{ .Values.vectorSync.queueMaxSize | quote }}
|
value: {{ .Values.vectorSync.queueMaxSize | quote }}
|
||||||
{{- end }}
|
{{- end }}
|
||||||
|
# Document Chunking (always set, used by vector sync processor)
|
||||||
|
- name: DOCUMENT_CHUNK_SIZE
|
||||||
|
value: {{ .Values.documentChunking.chunkSize | quote }}
|
||||||
|
- name: DOCUMENT_CHUNK_OVERLAP
|
||||||
|
value: {{ .Values.documentChunking.chunkOverlap | quote }}
|
||||||
# Qdrant Vector Database
|
# Qdrant Vector Database
|
||||||
{{- if eq .Values.qdrant.mode "network" }}
|
{{- if eq .Values.qdrant.mode "network" }}
|
||||||
# Network mode: Use dedicated Qdrant service
|
# Network mode: Use dedicated Qdrant service
|
||||||
|
|||||||
@@ -314,6 +314,20 @@ vectorSync:
|
|||||||
# Maximum queue size for documents pending indexing
|
# Maximum queue size for documents pending indexing
|
||||||
queueMaxSize: 10000
|
queueMaxSize: 10000
|
||||||
|
|
||||||
|
# Document Chunking Configuration
|
||||||
|
# Controls how documents are split into chunks before embedding
|
||||||
|
# Only relevant when vectorSync.enabled is true
|
||||||
|
documentChunking:
|
||||||
|
# Number of words per chunk (default: 512)
|
||||||
|
# Smaller chunks (256-384): Better for precise searches, more chunks to store
|
||||||
|
# Medium chunks (512-768): Balanced approach (recommended for most use cases)
|
||||||
|
# Larger chunks (1024+): Better for context, less precise matching
|
||||||
|
chunkSize: 512
|
||||||
|
# Number of overlapping words between chunks (default: 50)
|
||||||
|
# Recommended: 10-20% of chunkSize for context preservation across boundaries
|
||||||
|
# Must be less than chunkSize
|
||||||
|
chunkOverlap: 50
|
||||||
|
|
||||||
# Qdrant Vector Database Configuration
|
# Qdrant Vector Database Configuration
|
||||||
# Three deployment modes available:
|
# Three deployment modes available:
|
||||||
# 1. Local In-Memory: Fast, ephemeral, zero-config (mode: "memory")
|
# 1. Local In-Memory: Fast, ephemeral, zero-config (mode: "memory")
|
||||||
|
|||||||
Reference in New Issue
Block a user