feat(helm): Add document chunking configuration
Add support for configurable document chunking parameters to Helm chart to match docker-compose and application capabilities. Changes: 1. values.yaml: - Add documentChunking section with chunkSize (512) and chunkOverlap (50) - Include comprehensive comments explaining chunking strategies - Positioned between vectorSync and qdrant sections 2. templates/deployment.yaml: - Add DOCUMENT_CHUNK_SIZE and DOCUMENT_CHUNK_OVERLAP env vars - Always set (not conditional), used by vector sync processor - Environment variables follow same pattern as config.py defaults 3. README.md: - Add documentChunking parameter table in Vector Search section - Document chunking strategies (small/medium/large chunks) - Explain overlap recommendations (10-20% of chunk size) Validation: - helm lint: Passes - helm template: Environment variables correctly generated - Custom values: Work as expected (tested with chunkSize=1024) - Always present: Not conditional on vectorSync.enabled This maintains feature parity between Helm and docker-compose deployments, allowing users to tune chunking for their embedding models and use cases. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
@@ -219,6 +219,19 @@ Enable semantic search capabilities by deploying a vector database (Qdrant) and
|
||||
| `vectorSync.processorWorkers` | Number of concurrent processor workers | `3` |
|
||||
| `vectorSync.queueMaxSize` | Maximum queue size for pending documents | `10000` |
|
||||
|
||||
**Document Chunking Configuration:**
|
||||
|
||||
| Parameter | Description | Default |
|
||||
|-----------|-------------|---------|
|
||||
| `documentChunking.chunkSize` | Number of words per chunk for embedding | `512` |
|
||||
| `documentChunking.chunkOverlap` | Number of overlapping words between chunks | `50` |
|
||||
|
||||
**Chunking Strategy:**
|
||||
- **Small chunks (256-384)**: Better precision for searches, more storage overhead
|
||||
- **Medium chunks (512-768)**: Balanced approach (recommended for most use cases)
|
||||
- **Large chunks (1024+)**: Better context preservation, less precise matching
|
||||
- **Overlap**: Should be 10-20% of chunk size to preserve context across boundaries
|
||||
|
||||
**Qdrant Vector Database:**
|
||||
|
||||
Qdrant is deployed as a subchart when `qdrant.enabled` is `true`. All configuration values are passed through to the [qdrant/qdrant](https://github.com/qdrant/qdrant-helm) chart.
|
||||
|
||||
@@ -158,6 +158,11 @@ spec:
|
||||
- name: VECTOR_SYNC_QUEUE_MAX_SIZE
|
||||
value: {{ .Values.vectorSync.queueMaxSize | quote }}
|
||||
{{- end }}
|
||||
# Document Chunking (always set, used by vector sync processor)
|
||||
- name: DOCUMENT_CHUNK_SIZE
|
||||
value: {{ .Values.documentChunking.chunkSize | quote }}
|
||||
- name: DOCUMENT_CHUNK_OVERLAP
|
||||
value: {{ .Values.documentChunking.chunkOverlap | quote }}
|
||||
# Qdrant Vector Database
|
||||
{{- if eq .Values.qdrant.mode "network" }}
|
||||
# Network mode: Use dedicated Qdrant service
|
||||
|
||||
@@ -314,6 +314,20 @@ vectorSync:
|
||||
# Maximum queue size for documents pending indexing
|
||||
queueMaxSize: 10000
|
||||
|
||||
# Document Chunking Configuration
|
||||
# Controls how documents are split into chunks before embedding
|
||||
# Only relevant when vectorSync.enabled is true
|
||||
documentChunking:
|
||||
# Number of words per chunk (default: 512)
|
||||
# Smaller chunks (256-384): Better for precise searches, more chunks to store
|
||||
# Medium chunks (512-768): Balanced approach (recommended for most use cases)
|
||||
# Larger chunks (1024+): Better for context, less precise matching
|
||||
chunkSize: 512
|
||||
# Number of overlapping words between chunks (default: 50)
|
||||
# Recommended: 10-20% of chunkSize for context preservation across boundaries
|
||||
# Must be less than chunkSize
|
||||
chunkOverlap: 50
|
||||
|
||||
# Qdrant Vector Database Configuration
|
||||
# Three deployment modes available:
|
||||
# 1. Local In-Memory: Fast, ephemeral, zero-config (mode: "memory")
|
||||
|
||||
Reference in New Issue
Block a user