最新消息:雨落星辰是一个专注网站SEO优化、网站SEO诊断、搜索引擎研究、网络营销推广、网站策划运营及站长类的自媒体原创博客

google cloud platform - Why did my Airflow DAGs suddenly fail with "PythonVirtualenvOperator only supports functions fo

programmeradmin2浏览0评论

Issue Description I have multiple Airflow DAGs using a custom operator. All of these DAGs were working perfectly fine a few hours ago, but now they're all showing "Broken dag" status with the following error:

File "/opt/python3.11/lib/python3.11/site-packages/airflow/models/baseoperator.py", line 429, in apply_defaults
result = func(self, **kwargs, default_args=default_args)
File "/opt/python3.11/lib/python3.11/site-packages/airflow/operators/python.py", line 346, in __init__ 
raise AirflowException("PythonVirtualenvOperator only supports functions for python_callable arg")
airflow.exceptions.AirflowException: PythonVirtualenvOperator only supports functions for python_callable arg

My Operator Code Here's the relevant section of my DAG where I'm using the PythonVirtualenvOperator (with generic module/function names):

sys.path.append(f'{DAGS_FOLDER}/src/')
from data_transfer import transfer_function

transfer_task = PythonVirtualenvOperator(
    task_id='transfer_data',
    python_callable=transfer_function,
    requirements=['paramiko', 'google-cloud-secret-manager', 'google-cloud-logging', 'google-cloud-storage'],
    python_version='3.11',
    use_dill=True,
    system_site_packages=True,
    op_args=()
)

Environment Details

Airflow is running on Google Cloud Composer The imported function is definitely a function (not a module or class) No code changes were made between when it was working and when it broke This issue affects multiple DAGs that use this operator pattern

What I've Tried

Confirmed that the imported function exists and is a callable function Checked for any obvious syntax errors in my DAG files Looked through cloud logs but haven't found any specific errors

Questions

  1. What could cause the PythonVirtualenvOperator to suddenly start rejecting a function that was working fine before?
  2. How can I debug what's happening with the import of my function?
  3. Are there any known issues with PythonVirtualenvOperator in recent Airflow/Composer versions?
  4. What commands can I use to check if there were any recent updates to my Composer environment?

与本文相关的文章

发布评论

评论列表(0)

  1. 暂无评论