# oceanbase-database

Part of **OCEANBASE**

# OceanBase Database Utility Operations Troubleshooting Guide

## Problem Index

| Problem | Symptom | Severity | Solution Summary |
|--------|--------|---------|------------------|
| Connection Timeout | Error message: `OB0001: Connection timeout` | High | Verify network settings and instance accessibility |
| Deadlock Detected | Error message: `OB0002: Deadlock detected` | Medium | Roll back transaction and retry with exponential backoff |
| Transaction Resource Exhaustion | Error message: `OB0003: Transaction failed due to resource exhaustion` | High | Increase memory allocation or reduce concurrent load |

## Problem Details

### Problem 1: Connection Timeout (OB0001)

**Symptoms**
- Error message: `OB0001: Connection timeout. Verify network settings and ensure the instance is accessible.`
- Behavior: Client fails to establish a connection to the OceanBase instance; requests hang or fail immediately
- Context: Typically occurs during initial connection attempts, after network changes, or when security groups/firewalls block traffic

**Root Cause**
- The client cannot reach the OceanBase instance due to network misconfiguration, firewall rules, incorrect endpoint, or the instance being stopped or unresponsive.
- Common triggers include VPC misrouting, security group restrictions, or DNS resolution failures.

**Solution**
1. Confirm the OceanBase instance is in **Running** state via the console:
   - Navigate to **Console > Database Services > Instance Management**
2. Validate network connectivity from the client host:
   ```bash
   telnet <oceanbase-endpoint> <port>
   ```
   Replace `<oceanbase-endpoint>` and `<port>` with your instance’s actual values.
3. Check security group rules to ensure inbound traffic is allowed on the database port (default: 2883).
4. If using private endpoints, verify that the client resides in the same VPC or has proper peering/VPC sharing configured.

**Verification**
- Run a simple connection test using the OceanBase client:
  ```bash
  obclient -h <endpoint> -u <user>@<tenant> -P <port> -p'<password>'
  ```
- Successful login confirms the issue is resolved.

---

### Problem 2: Deadlock Detected (OB0002)

**Symptoms**
- Error message: `OB0002: Deadlock detected. Rollback the transaction and retry with backoff strategy.`
- Behavior: Application receives a deadlock error during concurrent DML operations (e.g., UPDATE, DELETE)
- Context: Occurs under high concurrency when multiple transactions hold locks on resources needed by others

**Root Cause**
- Two or more transactions are waiting for each other to release row or table locks, creating a circular dependency.
- OceanBase’s deadlock detector identifies this condition and aborts one transaction to break the cycle.

**Solution**
1. Catch the `OB0002` error in your application logic.
2. Roll back the current transaction explicitly:
   ```sql
   ROLLBACK;
   ```
3. Implement an exponential backoff retry mechanism before reissuing the transaction:
   ```python
   import time
   import random

   max_retries = 5
   base_delay = 0.1  # seconds

   for attempt in range(max_retries):
       try:
           # Execute your transaction here
           break
       except Exception as e:
           if "OB0002" in str(e):
               delay = base_delay * (2 ** attempt) + random.uniform(0, 0.1)
               time.sleep(delay)
           else:
               raise
   ```

**Verification**
- Monitor application logs for successful transaction completion after retries.
- Use the OceanBase console to check **Detected Exceptions** tab—no new `OB0002` entries should appear during normal operation.

---

### Problem 3: Transaction Resource Exhaustion (OB0003)

**Symptoms**
- Error message: `OB0003: Transaction failed due to resource exhaustion. Increase memory allocation or reduce load.`
- Behavior: Transactions fail intermittently under heavy load; error appears during large DML batches or complex queries
- Context: Common during bulk data ingestion, ETL jobs, or sudden traffic spikes

**Root Cause**
- The tenant’s allocated memory is insufficient to handle the current transaction workload.
- OceanBase enforces per-tenant memory limits; exceeding them causes transaction aborts.

**Solution**
1. Access the OceanBase console:
   - Go to **Console > Database Services > Instance Management > Exception Handling**
2. Review current tenant memory usage in the **Monitoring** section.
3. Increase tenant memory quota:
   - Navigate to **Tenant Management > [Your Tenant] > Configuration**
   - Adjust the `memory_limit` parameter (e.g., from 8G to 16G)
   ```sql
   ALTER TENANT <tenant_name> SET RESOURCE_POOL_LIST = ('new_pool_with_more_memory');
   ```
   > Note: Requires administrative privileges and may involve creating a new resource pool.
4. Alternatively, reduce transaction batch size or add rate limiting in the application.

**Verification**
- After applying changes, run the same workload and confirm no `OB0003` errors occur.
- Check tenant memory usage metrics—should remain below 85% during peak load.

## FAQ

**Q: How do I check if my OceanBase instance is healthy?**  
A: In the OceanBase console, go to **Instance Management** and verify the instance status is **Running**. Additionally, review the **Exception Handling** section for any active Critical or Warning-level exceptions.

**Q: What permissions are required to view or resolve exceptions?**  
A: You need administrative privileges on the OceanBase instance, typically granted via the `SYS` tenant or a user with `ALL PRIVILEGES` on the target tenant. API access requires a valid bearer token with sufficient scope.

**Q: How can I enable detailed logging for debugging database operations?**  
A: OceanBase does not expose direct log configuration to end users via the standard interface. However, you can capture diagnostic information through the **Exception Handling** console, which includes stack traces and timestamps. For deeper diagnostics, contact support with the exception ID.

**Q: Are deadlocks automatically resolved by OceanBase?**  
A: Yes. OceanBase includes a built-in deadlock detector that automatically identifies and resolves deadlocks by rolling back one of the involved transactions (returning error `OB0002`). Applications must handle this error and retry.

**Q: What is the maximum number of connections allowed per instance?**  
A: The quota limit is 1000 concurrent connections per instance. Exceeding this may cause new connection attempts to fail. Monitor active connections via the console and optimize connection pooling in your application if needed.