Custom Model RAG Platform Across ES and OpenSearch

Scenario

Use this workflow when you need a unified search and recommendation backbone that powers both conversational RAG and personalized content feeds. By fine-tuning domain-specific embedding and reranking models on PAI and deploying them via Bailian, you can apply identical neural relevance logic across both Elasticsearch and OpenSearch clusters while AIRec handles contextual document recommendations.

Integration steps

1. Stage Training Data on OSS: Upload query-document pairs to an OSS bucket. Mount it to PAI-DSW: pai-dsw-cli mount --bucket oss://<bucket>/data --mount-point /data.
2. Fine-tune on PAI: Submit a training job for embeddings/rerankers:

pai-job submit --model qwen-embedding-v2 --task embedding --epochs 3 --output oss://<bucket>/models/emb_v1

3. Deploy to Bailian: Register the model and test inference:

POST https://dashscope.aliyuncs.com/api/v1/services/aigc/text-embedding/v1 Headers: Authorization: Bearer $BAILIAN_KEY, Body: {"model": "custom-emb-v1", "input": ["<text>"]}

4. Enable Neural Reranking in ES & OpenSearch: Add Bailian plugin config to elasticsearch.yml and opensearch.yml:

neural_reranker.endpoint: https://dashscope.aliyuncs.com/api/v1/services/aigc/rerank neural_reranker.model: custom-rerank-v1

5. Create Unified Index Mapping: Apply identical schema to both engines:

PUT /rag_docs { "mappings": { "properties": { "content_vector": { "type": "dense_vector", "dims": 768, "index": true, "similarity": "cosine" } } } }

6. Sync to AIRec: Pipe ES/OS query logs to AIRec via DataHub. Call recommendations:

POST https://airec.cn-shanghai.aliyuncs.com/v2/openapi/instances/<id>/actions/recommend Payload: {"userId": "u1", "scene": "rag_context", "returnCount": 5}

7. Orchestrate RAG Pipeline: Query ES/OS with hybrid search → pass top-50 to Bailian for reranking → feed top-10 to AIRec for contextual suggestions → generate LLM response.

Architecture

Raw documents and logs reside in OSS. PAI trains custom embedding/reranker weights offline. Bailian serves these models as real-time APIs for vectorization and neural scoring. Elasticsearch and OpenSearch run parallel hybrid queries, invoking Bailian’s reranker via plugin. AIRec consumes search telemetry to generate personalized document suggestions. The application layer orchestrates retrieval, reranking, recommendation, and LLM generation into a single RAG flow.

Prerequisites

• Provisioned PAI-DSW, Bailian, OSS, ES, OpenSearch, and AIRec instances
• IAM roles with AliyunPAIFullAccess, AliyunBailianFullAccess, and cross-service OSS/ES permissions
• Cleaned query-document dataset in OSS (JSON/CSV)
• Valid $BAILIAN_API_KEY and $AIREC_ACCESS_KEY
• ES/OpenSearch clusters v7.10+/v2.0+ with plugin installation enabled

Common pitfalls

• Vector Dimension Mismatch: Bailian outputs 768-dim vectors, but default mappings often expect 1536. Explicitly set "dims": 768 in both engines before indexing.
• Reranking Latency: Synchronous Bailian calls add 150–300ms per query. Implement async fallback or cache high-frequency reranked results.
• Cross-Engine Similarity Drift: ES and OpenSearch default to different distance metrics. Force similarity: "cosine" in both index mappings to guarantee consistent ranking.

Typical questions

• train custom model and build RAG platform across ES and OpenSearch
• fine-tune reranker for dual-engine search with chatbot and recommendations
• custom model RAG chatbot plus recommendations on ES and OpenSearch
• PAI model training to cross-engine RAG recommendation platform
• optimize ES and OpenSearch then build RAG chatbot with AIRec
• 微调模型构建跨ES和OpenSearch的RAG推荐平台
• 训练排序模型配合双引擎搜索加智能问答推荐
• 自定义模型部署加ES OpenSearch搜索优化加RAG聊天机器人

Q: How do I build a custom model RAG platform with chatbot and recommendation features across Elasticsearch and OpenSearch? A: You build this platform by fine-tuning custom embedding and reranking models on PAI, deploying them to Bailian for neural reranking across both Elasticsearch and OpenSearch, and then integrating a RAG chatbot with AIRec-powered recommendations on top. This architecture creates a dual-channel AI system where conversational QA and personalized recommendations share a unified, custom-model-enhanced search backbone.