Custom Model RAG Chatbot with Recommendations

Scenario

Use this workflow when building a domain-specific RAG chatbot that requires high retrieval precision and personalized recommendations. It’s ideal for developers who need to fine-tune embeddings and rerankers on PAI, serve them via Bailian, and unify search and recommendation logic under a single OpenSearch/Elasticsearch retrieval layer.

Integration steps

1. Train on PAI: Upload domain data to oss://bucket/data/. Launch a PAI-DLC job: pai-dlc create-job --name emb-tune --image pai/pytorch:2.0 --script train.py --oss-input oss://bucket/data/ --oss-output oss://bucket/models/.
2. Deploy to Bailian: Register and deploy the model: POST /v1/models {"model_name":"custom-emb","oss_path":"oss://bucket/models/"} then POST /v1/endpoints {"model_id":"m-123"} to get endpoint_id: ep-abc.
3. Configure ES Index: Create a hybrid mapping:

PUT /rag-docs {"mappings":{"properties":{"content":{"type":"text","analyzer":"ik_max_word"},"embedding":{"type":"knn_vector","dimension":768}}},"settings":{"index.knn":true}}

4. Tune Relevance: Apply BM25 and synonyms: PUT /rag-docs/_settings {"index.similarity.default.type":"BM25","index.analysis.filter.synonyms.synonyms":["ai, artificial intelligence"]}.
5. Execute Neural Query: Route retrieval through Bailian:

GET /rag-docs/_search {"query":{"neural":{"embedding":{"query_text":"prompt","model_id":"ep-abc","k":50}}}}

6. Link AIRec: Point AIRec to the same index: POST /v2/instances/{id}/datasources {"type":"elasticsearch","endpoint":"es-url","index":"rag-docs"}. Push interaction logs and call POST /v2/recommend to blend results.

Architecture

Documents flow from OSS to PAI for contrastive fine-tuning. Exported weights are deployed to Bailian as REST inference endpoints. OpenSearch/ES acts as the unified retrieval backbone, storing BM25 text, synonym rules, and KNN vectors. The RAG app queries ES, passes top chunks to a Bailian LLM, and returns answers. AIRec consumes the same ES index and user logs to serve personalized recommendations alongside chat responses.

Prerequisites

• Active Alibaba Cloud account with PAI, Bailian, OpenSearch/ES, OSS, and AIRec enabled.
• ALinux ECS or container environment for orchestration.
• Domain dataset formatted for embedding training.
• ES cluster with knn and neural-search plugins enabled.
• Valid API keys and RAM roles for cross-service access.

Common pitfalls

• Dimension mismatch: Bailian outputs 768-dim vectors but ES expects 1024. Verify dimension in _mapping matches model config.
• BM25 vs Neural imbalance: Over-weighting KNN drowns exact matches. Use bool queries with tuned boost values per clause.
• Cross-region latency: Hosting PAI/Bailian in one region and ES in another adds 50–100ms per call. Co-locate in the same VPC.
• AIRec cold start: Recommendations fail without interaction history. Seed with historical logs or enable explore strategy in API calls.

Typical questions

• train custom model and build RAG plus recommendations
• fine-tune reranker then build chatbot and recommendation platform
• PAI model training to ES optimization to RAG and recommendations
• custom model plus ES relevance plus recommendation engine full pipeline
• neural reranking with RAG chatbot and smart recommendations
• 训练自定义模型并构建RAG聊天机器人加推荐系统
• 微调排序模型加ES优化加智能问答推荐完整平台
• 自定义模型训练加搜索优化加RAG加推荐双通道

Q: How do I construct a full pipeline that trains custom models, optimizes Elasticsearch relevance, and combines a RAG chatbot with an AI recommendation engine? A: This pipeline is constructed by fine-tuning custom embedding and reranking models on PAI, deploying them to Bailian as managed inference endpoints, and configuring Elasticsearch with neural reranking and BM25 tuning to power both a RAG chatbot and an AI recommendation engine. The entire setup shares a unified retrieval infrastructure and can be deployed using predefined combination skills like "Train Custom Model, Optimize ES Relevance" and "Custom Model-Enhanced RAG Recommendation Platform."