# opensearch-abtest

Part of **OPENSEARCH**

# OpenSearch A/B Testing and Evaluation

## Capabilities Overview

| Sub-capability | Calling Mode | Description |
|----------------|--------------|-------------|
| Manage A/B Tests | Synchronous | Create, update, delete, and list A/B testing experiments, groups, and scenes. |
| Create AB Test Experiment | Synchronous | Set up AB testing experiments for search features. |
| Get AB Test Scene Details | Synchronous | Retrieve detailed information about AB test scenes. |
| Create A/B Test | Synchronous | Create A/B test experiments, groups, or scenes. |
| Create A/B Test Group | Synchronous | Define groups for A/B testing experiments in OpenSearch. |
| Create A/B Test Scene | Synchronous | Configure scenes for A/B testing scenarios. |
| Retrieve A/B Test Configuration | Synchronous | Get details about existing A/B test experiment configurations. |
| Create A/B Experiment | Synchronous | Create A/B test experiments for search analysis. |
| Query Statistics | Synchronous | Retrieve statistical logs or reports. |
| Compute Aggregated Statistics | Synchronous | Calculate statistical aggregates using the aggregate clause. |
| Compute Aggregations | Synchronous | Perform aggregation operations using the aggs clause. |

## API Calling Patterns

### Authentication
Use **Bearer Token** authentication as the primary method.

- Include the header: `Authorization: Bearer <your_api_key>`
- Store your API key in the environment variable: `DASHSCOPE_API_KEY`
- Example: `export DASHSCOPE_API_KEY=sk-xxxxxx`

While other authentication methods may exist, Bearer Token is consistently used across all documented endpoints and is recommended for simplicity and compatibility.

### Service Endpoint (Endpoint)
APIs use region-specific endpoints with the following pattern:

`https://opensearch.{region}.aliyuncs.com/v4/openapi/...`

Common regions include:
- `cn-hangzhou` (China)
- `cn-shanghai` (International)
- `ap-southeast-1` (International)

Some management APIs also accept global endpoints like `https://api.aliyun.com/v4/openapi/...` or `https://api.alibabacloud.com/v4/openapi/...`, but regional endpoints are preferred for lower latency.

### Synchronous API Pattern
All A/B testing and evaluation APIs in OpenSearch follow a **synchronous** calling pattern:

1. **Send a single HTTP request** (GET, POST, PUT, or DELETE) to the appropriate endpoint with required path/query parameters and headers.
2. **Include the `Authorization: Bearer $DASHSCOPE_API_KEY` header** in every request.
3. **Receive an immediate JSON response** containing either:
   - A `result` field with the requested data (e.g., experiment details, list of groups)
   - An empty `result` object for deletion operations
   - Error details if the request fails
4. **Parse the `requestId`** from the response for logging or support purposes.
5. **Handle standard HTTP status codes** (200 for success, 4xx/5xx for errors).

No polling or streaming is required—each call completes within the same request-response cycle.

## Parameter Reference

### Manage A/B Tests (Experiments)

| Parameter | Type | Required | Default | Constraints | Description |
|----------|------|--------|--------|------------|-------------|
| appGroupIdentity | string | Yes | — | — | The application name. |
| sceneId | integer | Yes | — | — | The scenario ID. |
| groupId | integer | Yes | — | — | The group ID. |
| experimentId | integer | Yes (for GET/PUT/DELETE) | — | — | The experiment ID. |
| dryRun | boolean | No | false | one of: true, false | Specifies whether to perform a dry run. Valid values: true (checks only), false (executes). |
| body | object | No | — | — | The request body containing experiment parameters (e.g., name, traffic, online status). |

### Manage A/B Test Groups

| Parameter | Type | Required | Default | Constraints | Description |
|----------|------|--------|--------|------------|-------------|
| appGroupIdentity | string | Yes | — | — | The name of the application. |
| sceneId | integer | Yes | — | — | The scenario ID. |
| groupId | integer | Yes (for GET/PUT) | — | — | The ID of the group. |
| dryRun | boolean | No | false | true or false | Specifies whether to check validity without creating/updating. |
| body | object | No | — | — | The request body containing group details (e.g., name, status). |

### Manage A/B Test Scenes

| Parameter | Type | Required | Default | Constraints | Description |
|----------|------|--------|--------|------------|-------------|
| appGroupIdentity | string | Yes | — | — | The name of the application group. |
| sceneId | integer | Yes (for GET/PUT) | — | — | The ID of the scene. |
| dryRun | boolean | No | false | one of: true, false | Specifies whether to validate only the request parameters. |
| body | object | No | — | — | The A/B test scene configuration (e.g., name, values, status). |

### Query Statistics

| Parameter | Type | Required | Default | Constraints | Description |
|----------|------|--------|--------|------------|-------------|
| appGroupIdentity | string | Yes | — | — | The application name. |
| moduleName | string | Yes | — | one of: hot, error, slow-log, app, app-query, abtest, suggest, hint, data-quality, es | The module or report type to query. |
| startTime | integer | No | current day 00:00:00 | UNIX timestamp | Start of time range (seconds since epoch). |
| stopTime / endTime | integer | No | current day 24:00:00 | UNIX timestamp | End of time range. |
| pageNumber | integer | No | 1 | range 1 to max | Page number for pagination. |
| pageSize | integer | No | 10 | max 100 | Number of entries per page. |
| query | string | No | — | format: 'k1:v1,k2:v2' | Query conditions (e.g., `experimentSerialNumber:123`). |
| columns | string | No | — | comma-separated field names | Fields to return (e.g., `pv,uv`). |
| sortBy | string | No | — | format: '-field' or 'field' | Sort order for log queries. |
| distinct | boolean | No | false | true or false | Enable distinct query for logs. |

### Compute Aggregated Statistics (aggregate clause)

| Parameter | Type | Required | Default | Constraints | Description |
|----------|------|--------|--------|------------|-------------|
| group_key | string | Yes | — | attribute field (INTEGER/STRING) | Field to group results by. |
| agg_fun | string | Yes | — | count(), sum(field), max(field), min(field), distinct_count(field) | Aggregation functions (separate with #). |
| range | string | No | — | format: n1~n2 | Numeric range for distribution analysis. |
| agg_filter | string | No | — | logical expression | Filter applied before aggregation. |
| agg_sampler_threshold | integer | No | — | Positive integer | Rank cutoff for full collection vs sampling. |
| agg_sampler_step | integer | No | — | Positive integer | Sampling step size for estimation. |
| max_group | integer | No | 1000 | — | Maximum number of groups returned. |

## Code Examples

### Create an A/B Test Experiment - curl - all

```bash
curl -X POST \
  'https://opensearch.cn-hangzhou.aliyuncs.com/v4/openapi/app-groups/my_app_group_name/scenes/20404/groups/13467/experiments' \
  -H 'Authorization: Bearer $DASHSCOPE_API_KEY' \
  -H 'Content-Type: application/json' \
  -d '{
    "name": "my_test_experiment",
    "traffic": 30,
    "online": true,
    "params": {
      "first_formula_name": "default",
      "qp_chain_name": "test_qp"
    }
  }'
```

### Delete an A/B Test Experiment - Python - all

```python
import requests
import os

url = "https://opensearch.cn-hangzhou.aliyuncs.com/v4/openapi/app-groups/{appGroupIdentity}/scenes/{sceneId}/groups/{groupId}/experiments/{experimentId}"
headers = {
    "Authorization": f"Bearer {os.getenv('DASHSCOPE_API_KEY')}",
    "Content-Type": "application/json"
}

response = requests.delete(url.format(
    appGroupIdentity="my_app",
    sceneId=20404,
    groupId=13467,
    experimentId=12889
))
print(response.json())
```

### List A/B Test Experiments - Python - all

```python
import requests
import os

url = "https://api.aliyun.com/v4/openapi/app-groups/{appGroupIdentity}/scenes/{sceneId}/groups/{groupId}/experiments"
headers = {
    "Authorization": f"Bearer {os.getenv('DASHSCOPE_API_KEY')}",
    "Content-Type": "application/json"
}

response = requests.get(url.format(
    appGroupIdentity="my_app",
    sceneId=20404,
    groupId=13467
))
print(response.json())
```

### Query A/B Test Statistical Report - curl - all

```bash
curl -X GET \
  'https://api.aliyun.com/v4/openapi/app-groups/my_app_group_name/statistic-report/abtest' \
  -H 'Authorization: Bearer $DASHSCOPE_API_KEY' \
  -H 'Content-Type: application/json' \
  -d '{
    "startTime": 1700000000,
    "endTime": 1700086400,
    "pageNumber": 1,
    "pageSize": 20,
    "columns": "pv,uv,conversion_rate",
    "query": "experimentSerialNumber:12889"
  }'
```

### Update A/B Test Whitelist (Fixed Flow Dividers) - curl - all

```bash
curl -X PUT \
  'https://api.aliyun.com/v4/openapi/app-groups/my_app_group_name/scenes/20404/groups/13467/experiments/12889/fixed-flow-dividers' \
  -H 'Authorization: Bearer $DASHSCOPE_API_KEY' \
  -H 'Content-Type: application/json' \
  -d '{
    "body": ["user_123", "user_456", "session_xyz"]
  }'
```

### Use aggregate Clause for Aggregated Statistics - plain - all

```text
aggregate=group_key:experiment_id,agg_fun:sum(conversion)#count(),max_group:100
```

### Use aggs Clause in JSON Request Body - json - all

```json
{
  "aggs": [
    {
      "group_key": "variant",
      "agg_fun": ["sum(revenue)", "count()", "distinct_count(user_id)"],
      "agg_filter": "timestamp > 1700000000"
    }
  ]
}
```

## Response Format

```json
{
  "requestId": "D77D0DAF-790D-F5F5-A9C0-133738165014",
  "result": {
    "created": 1588842080,
    "traffic": 30,
    "online": true,
    "name": "test1",
    "updated": 1588842080,
    "id": "12888",
    "params": {
      "first_formula_name": "default"
    }
  }
}
```

**Key Fields**:
- `requestId` — Unique identifier for the API request; useful for debugging and support.
- `result.id` — Unique ID of the created or retrieved resource (experiment, group, or scene).
- `result.name` — Human-readable name of the resource.
- `result.traffic` — Percentage of traffic allocated to the experiment (0–100).
- `result.online` — Boolean indicating whether the experiment is active.
- `result.created` / `result.updated` — UNIX timestamps for creation and last modification.
- `result.params` — Configuration parameters such as ranking formulas and query analysis chains.

## Error Handling

| Error Code | Description | Recommended Action |
|-----------|-------------|-------------------|
| 400 | Bad Request – The request parameters are invalid or missing. | Validate all required parameters (`appGroupIdentity`, `sceneId`, etc.) and ensure correct data types. |
| 401 | Unauthorized – Authentication failed. Check your API key or credentials. | Verify that `DASHSCOPE_API_KEY` is set correctly and has not expired. |
| 403 | Forbidden – Insufficient permissions to perform this operation. | Ensure your RAM user has the necessary OpenSearch permissions (e.g., `opensearch:CreateApp`, `opensearch:DescribeApp`). |
| 404 | Not Found – The specified app group, scene, group, or experiment does not exist. | Confirm that all identifiers exist in your OpenSearch instance and are spelled correctly. |
| 429 | Too Many Requests – Rate limit exceeded. | Reduce request frequency; implement exponential backoff for retries. |
| 500 | Internal Server Error – An unexpected error occurred on the server side. | Retry the request after a short delay. If persistent, contact Alibaba Cloud support with the `requestId`. |
| 503 | Service Unavailable – The service is temporarily unavailable. | Wait and retry later; check Alibaba Cloud status page for outages. |

### Rate Limits & Retry
- Most endpoints enforce **100 QPS per account or app group**.
- Some operations (e.g., `DescribeABTestExperiment`) explicitly state a **100 QPS limit**.
- When encountering `429` errors:
  - Implement **exponential backoff** (e.g., wait 1s, 2s, 4s, 8s…).
  - Respect the `Retry-After` header if provided (though not explicitly documented, it’s a common practice).
  - Distribute requests evenly over time to avoid bursts.

## Environment Requirements

- Set the environment variable: `export DASHSCOPE_API_KEY=your_api_key_here`
- Use any HTTP client library (e.g., `requests` in Python, built-in `fetch` or `axios` in JavaScript)
- For the `ListABTestFixedFlowDividers` API, ensure `dashscope>=1.14.0` if using the DashScope SDK (though direct HTTP calls are recommended for consistency)

## FAQ

Q: How do I authenticate my API requests to OpenSearch A/B Testing endpoints?
A: Include the header `Authorization: Bearer $DASHSCOPE_API_KEY` in every request, where `DASHSCOPE_API_KEY` is your API key stored as an environment variable.

Q: What is the difference between a scene, group, and experiment in A/B testing?
A: A **scene** represents a testing context (e.g., "homepage search"), a **group** organizes related experiments within a scene, and an **experiment** defines a specific variant with traffic allocation and configuration parameters.

Q: Can I update an experiment's traffic percentage without taking it offline?
A: Yes. Use the `UpdateABTestExperiment` API with `"online": true` and the new `traffic` value (0–100). Changes take effect immediately.

Q: How do I retrieve A/B test performance metrics like conversion rate or click-through rate?
A: Use the `ListStatisticReport` API with `moduleName=abtest` and specify relevant `columns` (e.g., `pv`, `uv`, `conversion_rate`) along with your `experimentSerialNumber` in the `query` parameter.

Q: Are there any limits on the number of experiments or groups I can create?
A: While no hard limits are documented, each API enforces rate limits (typically 100 QPS). Also, keep `max_group` ≤ 10,000 in aggregation queries to avoid memory issues.

## Pricing & Billing

### Billing Model
All A/B testing management and statistics APIs follow a **per-request** billing model. Each successful or failed API call counts as one request.

### Price Reference

| Tier | Input Price | Output Price |
|------|-------------|--------------|
| default | 0.001 / | 0.001 / |
| standard | 0.0001 / | 0.0001 / |

> Note: Prices vary slightly by endpoint (e.g., experiment creation vs. description), but all are billed per call.

### Free Tier
- **1000 free requests per month** for most endpoints
- Some endpoints (e.g., group creation) offer **100 free requests/month**

### Usage Limits
- **100 QPS per account or app group** for most APIs
- Aggregation accuracy is guaranteed only for up to **100,000 documents per partition**

### Billing Notes
- **Dry run requests (`dryRun=true`) are billed** the same as real operations.
- Failed requests (4xx/5xx) **are billed** unless explicitly stated otherwise (e.g., `DescribeABTestExperiment` notes that errors are not billed).
- Free tier resets monthly; usage beyond the quota is charged at the listed rates.