This project provides a script that helps to migrate data between Google Cloud Storage (GCS) buckets.
- Google Cloud SDK
- Python 3.x
- Google Cloud Storage Client for Python
-
Google Cloud SDK Installation:
Follow the instructions on the official page to install and initialize the Google Cloud SDK.
-
Python Installation:
Ensure that Python 3.x is installed on your system. If not, download it from the official website.
-
Google Cloud Storage Client Installation:
Install Google Cloud Storage Client for Python using pip:
pip install --upgrade google-cloud-storage
-
Authentication:
Ensure that you have the JSON key file for the Google Cloud Service Account and set it in your environment variables:
export GOOGLE_APPLICATION_CREDENTIALS="path_to_your_service_account_file.json"
-
Execute the following shell command to authenticate your gcloud CLI with Google Cloud:
gcloud auth login
-
Set your desired Google Cloud Project ID as the current project:
gcloud config set project your_project_id
-
Ensure that a Service Account with the necessary permissions to access and modify GCS buckets is available.
-
Set permissions and create keys for the Service Account:
gcloud iam service-accounts create your_service_account_name --description="your_description" --display-name="your_display_name" gcloud projects add-iam-policy-binding your_project_id --member=serviceAccount:your_service_account_name@your_project_id.iam.gserviceaccount.com --role=roles/storage.objectViewer gcloud iam service-accounts keys create ./your_key_name.json --iam-account your_service_account_name@your_project_id.iam.gserviceaccount.com
-
Execute the Python script to perform the bucket data migration:
python bucket_migration.py
- The script will perform the following operations:
- List all the files in the source bucket.
- Download the files from the source bucket to a temporary location.
- Upload the files from the temporary location to the destination bucket.
Ensure to review and modify the bucket_migration.py
script to set appropriate source and destination bucket names and other configurations as per your use-case.