Friday 11 February 2022

Defining 'S3ToGCSOperator' in GCP Airflow DAG.

 I have the following operator that I want to use in this DAG.  3ToGCSOperator

transfer_data_s3_to_gcs = S3ToGCSOperator(
task_id='data_transfer_gcs_to_gcs',


However I need to define the import. 

Which is 

from airflow.providers.google.cloud.operators.s3_to_gcs import S3ToGCSOperator


What you need to do is your Environment in GCP and navigate to the ‘PYPI PACKAGES’ TAB.
And add the package https://airflow.apache.org/docs/apache-airflow-providers-google/stable/index.html

apache-airflow-providers-google

However I then got this error. 


UPDATE operation on this environment failed 5 minutes ago with the following error message:

Failed to install PyPI packages. apache-airflow-providers-google 6.3.0 has requirement google-ads<14.0.1,>=12.0.0, but you have google-ads 7.0.0.

 Check the Cloud Build log at https://console.cloud.google.com/cloud-build/builds/%%?project=4%% for details. For detailed instructions see https://cloud.google.com/composer/docs/troubleshooting-package-installatio


So what you need to do first in ‘PYPI PACKAGES’ is install oogle-ads. Version =  >=12.0.0,<14.0.1.


ModuleNotFoundError: No module named 'airflow.providers.google.cloud.operators.s3_to_gcs'

And then you'll be able to do it. 

No comments: