ThisCalculator

Everyday

Troubleshoot AI Model Compatibility on Ascend

Updated · Standard formula, shown below

Troubleshoot AI Model Compatibility on Ascend

This guide helps you diagnose and fix compatibility and performance issues when running AI models on Huawei Ascend hardware (like the 910B or 310P). You’ll learn step-by-step methods to check your environment, convert models correctly, and optimize inference speed using the Ascend toolkit and CANN.

What You’ll Need

  • Access to an Ascend device (e.g., Atlas 800/900 server or Atlas 200 DK).
  • The CANN toolkit installed (version 5.1 or later recommended).
  • The Ascend TensorFlow or PyTorch adapter (e.g., torch_npu).
  • A model file in a common format (ONNX, TensorFlow SavedModel, PyTorch .pt, or MindSpore).
  • Python 3.7+ with pip and the ability to install packages.
  • Basic familiarity with the command line and model training/inference concepts.

Step-by-Step

  1. Verify device visibility. Run npu-smi info to list your Ascend devices. If nothing appears, check that the driver is loaded (lsmod | grep drv) and that the device is powered on. If the device is not visible, reinstall the driver or check the physical connection.
  2. Check CANN and adapter versions. Run python -c "import torch; import torch_npu; print(torch.__version__, torch_npu.__version__)" (for PyTorch) or python -c "import tensorflow as tf; print(tf.__version__)". Ensure the versions match your CANN version. Mismatches are a common cause of import errors or crashes. Refer to the official compatibility matrix.
  3. Test with a simple model. Run a minimal inference script (e.g., a single-layer network) to confirm the basic stack works. If this fails, the issue is environmental, not model-specific.
  4. Convert your model to a supported format. Ascend prefers ONNX or MindSpore. For PyTorch, export to ONNX using torch.onnx.export. For TensorFlow, use the SavedModel format. Use the atc tool (Ascend Tensor Compiler) to convert to an .om file: atc --model=model.onnx --framework=5 --output=model --soc_version=Ascend910B. Replace the soc_version with your chip type (check with npu-smi info).
  5. Check operator support. Some operators (e.g., custom ops, certain attention variants) may not be supported. Use the msopgen or atc log to see which ops fail. If an op is unsupported, you may need to replace it with a supported equivalent or use a different model version.
  6. Run inference and measure performance. Use the benchmark tool from CANN (e.g., benchmark --om=model.om --batchSize=1 --loopCount=100) to get throughput (FPS) and latency. Compare against expected performance for your chip.
  7. Optimize for performance. If performance is lower than expected, try:
    • Increasing batch size to better utilize the AI Core.
    • Enabling mixed precision (FP16) if your model supports it.
    • Using the --insert_op_conf to enable AOE (Ascend Optimized Engine) for auto-tuning.
    • Checking if the model is memory-bound; reduce memory usage by using smaller input sizes or model pruning.
  8. Check logs for errors. Look at the CANN logs (typically in ~/ascend/log or /var/log/npu). Search for “ERROR” or “FAILED” to pinpoint issues.

Common Mistakes

  • Using wrong soc_version. Specifying the wrong chip type in atc can cause conversion failures or runtime errors. Always verify with npu-smi info.
  • Ignoring the CANN version compatibility. Using a newer CANN with an older adapter (or vice versa) often leads to import errors or undefined symbols.
  • Assuming all ONNX ops are supported. Many ONNX ops are not natively supported on Ascend; you need to check and possibly rewrite the model.
  • Forgetting to set environment variables. On some setups, you must source the CANN environment script (e.g., source /usr/local/Ascend/ascend-toolkit/set_env.sh) before running Python or atc.
  • Not using the correct input format. Ascend often expects NCHW format, while some frameworks default to NHWC. Mismatches cause shape errors.

Tips

  • Use the msopgen tool to generate a model that only uses supported ops from the start.
  • For PyTorch, use the torch_npu plugin and call model.npu() to move the model to Ascend, but prefer exporting to ONNX for production.
  • Profile your model with the Ascend Profiler to identify bottlenecks (e.g., memory copy, operator execution).
  • Keep the CANN toolkit updated to the latest patch for bug fixes and performance improvements.
  • If you have a large model, consider using model parallel or distributed inference across multiple NPUs.

Troubleshooting

  • Model conversion fails with “Unsupported op”. Identify the op from the log. Try to replace it with a supported alternative (e.g., use a standard attention instead of a custom one). If the op is critical, you may need to use a different model architecture.
  • Runtime error: “Device memory exhausted”. Reduce batch size, use FP16, or optimize the model to use less memory. Also check if other processes are using the NPU.
  • Inference is very slow. Check if the model is using the CPU instead of the NPU. Ensure you have called model.npu() or used the correct device context. Also check if the model is being recompiled each run; use the .om file directly.
  • Import error: “libascendcl.so not found”. The CANN environment is not set. Source the set_env.sh script or add the library path to LD_LIBRARY_PATH.
  • Results are incorrect (NaN or wrong values). Check for numerical precision issues. Try using FP32 instead of FP16, or verify that the model weights are correctly loaded. Also ensure that the input data is normalized the same way as during training.

Frequently Asked Questions

  1. Can I run any PyTorch model on Ascend? Not directly. You need to export to ONNX or use the torch_npu plugin, and even then, some ops may not be supported. You may need to modify the model.
  2. What is the difference between .om and .onnx files? .onnx is a general model format, while .om is the optimized format for Ascend. The .om file is produced by the atc compiler and is required for efficient inference on the NPU.
  3. How do I check if my model is running on the NPU? Use npu-smi info to see the NPU utilization. If it’s near 0% during inference, your model is probably running on the CPU.
  4. Why does my model compile every time I run it? If you are using a framework model (not .om), the compiler may re-optimize the graph each time. To avoid this, convert to .om and load it directly.
  5. Can I use TensorFlow models on Ascend? Yes, TensorFlow models can be converted to .om using the atc tool with the SavedModel format. However, you may need to freeze the graph first.
  6. What is AOE and should I use it? AOE (Ascend Optimized Engine) automatically tunes your model for better performance. It can be enabled during atc conversion and is recommended for production deployments.

Related

Method: the standard formula for this calculation, rounded as each result states. See how the calculators work or report an error.