DaaS / Products / ML Training Pipeline End-to-End Monitoring

ML Training Pipeline End-to-End Monitoring

A data scientist runs ML training jobs on PAI that read/write large datasets in RDS. When training is slow or fails, they need to monitor PAI job metrics (GPU utilization, training logs) alongside RDS performance (slow queries, database CPU) to pinpoint whether the bottleneck is in the compute layer or the data layer.

Products involved

Scenario

A data scientist runs ML training jobs on PAI that read/write large datasets in RDS. When training is slow or fails, they need to monitor PAI job metrics (GPU utilization, training logs) alongside RDS performance (slow queries, database CPU) to pinpoint whether the bottleneck is in the compute layer or the data layer.

How the products combine

  1. pai · pai-monitor-jobs — Platform for AI (PAI) — Monitor and debug AI jobs

    2. See pai/pai-monitor-jobs.

  2. rds · rds-monitor-performance — ApsaraDB RDS — Monitor and analyze database performance metrics

    3. See rds/rds-monitor-performance.

Typical questions