Deploy Production Stack Then Debug DB Connectivity

Scenario

Use this workflow when provisioning a production web stack via Terraform and immediately troubleshooting application-level database connection failures between ALinux-based ECS instances and ApsaraDB RDS. It bridges infrastructure-as-code deployment with targeted network and credential validation to rapidly restore secure, compliant connectivity.

Integration steps

1. Provision Infrastructure: Run terraform init and terraform apply. Define alicloud_vpc, alicloud_security_group, alicloud_instance (ALinux), alicloud_db_instance (RDS), alicloud_oss_bucket, and alicloud_slb with alicloud_ssl_certificate from CAS.
2. Configure RDS Access: In main.tf, set security_ips to the ECS VPC CIDR (e.g., 10.0.0.0/16) and provision credentials: resource "alicloud_db_account" "app_user" { instance_id = alicloud_db_instance.main.id; name = "app_user"; password = "SecurePass123!" }.
3. Deploy App to ECS: Use remote-exec to install dependencies on ALinux: yum install -y mysql-connector-python python3-pip. Deploy your application binary.
4. Validate Network Path: SSH into ECS and run nc -zv <rds-endpoint>.rds.aliyuncs.com 3306. Success confirms L3/L4 reachability.
5. Inspect Security Groups: If nc hangs, verify ECS outbound and RDS inbound rules allow TCP 3306. Check VPC route tables for missing NAT/IGW entries.
6. Test Authentication: Run mysql -h <rds-endpoint> -u app_user -p -P 3306. If it fails, verify account privileges and ensure the RDS whitelist matches the ECS private IP.
7. Verify OS Drivers: If the app throws ImportError: libmysqlclient.so.21, run ldd $(python3 -c "import mysql.connector; print(mysql.connector.__file__)") on ALinux and install missing libs via yum install mysql-community-libs-compat.

Architecture

Terraform orchestrates the lifecycle of all resources. User traffic hits the SLB, which terminates HTTPS using CAS-managed certificates. The SLB forwards requests to the ECS cluster running ALinux. ECS instances query the RDS backend over the private VPC network for dynamic data, while static assets are offloaded to OSS. All components are bound by shared security groups and IAM roles.

Prerequisites

• Alibaba Cloud RAM roles: AliyunVPCFullAccess, AliyunECSFullAccess, AliyunRDSFullAccess, AliyunSLBFullAccess.
• Terraform CLI (v1.5+) with alicloud provider configured via ALICLOUD_ACCESS_KEY and ALICLOUD_SECRET_KEY.
• Valid domain pointing to SLB VIP for CAS certificate issuance.
• ALinux 3.x image ID and target RDS engine (MySQL 8.0/PostgreSQL 14).

Common pitfalls

• RDS Whitelist Mismatch: Adding the ECS public IP instead of its private VPC IP to security_ips, causing timeouts over the internal network.
• Restrictive Egress Rules: Default ALinux security groups often block outbound traffic; explicitly allow TCP 3306/5432 to the RDS CIDR.
• Missing Client Libraries: ALinux minimal images lack native DB drivers, causing silent ImportError or ModuleNotFoundError in application runtimes.
• Terraform State Drift: Manual console edits to RDS parameters or SLB listeners break terraform plan and cause inconsistent connectivity during updates.

Typical questions

• terraform deploy production stack then debug ECS to RDS connection
• provision full web stack and troubleshoot database connectivity
• deploy production infra with Terraform then fix app database connection refused
• ECS cannot connect to RDS after terraform deployment
• terraform部署生产环境后排查ECS连接RDS失败
• provision complete stack then debug database permission denied on ECS
• deploy secure web stack with RDS then troubleshoot connectivity issues
• terraform create VPC ECS RDS SSL then diagnose connection problems

Q: How do I troubleshoot database connectivity issues between ECS and RDS after deploying a production stack with Terraform? A: You can resolve these connection failures by applying the dedicated ECS App Database Connectivity debugging workflow after completing your Terraform-based infrastructure provisioning. This approach addresses application-level connectivity issues between deployed ECS instances and the RDS backend within a complete production web stack that includes VPC, OSS, and SSL-enabled SLB components.