# Storage ![](https://1374779285-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-MFW3x2aqEO8GF2kr3VU%2Fuploads%2FB3Wb8OsAaZCrOrA2eqiJ%2Fimage.png?alt=media\&token=d66ba4b2-200f-47ff-8acc-0cdd05cac136) ## Cloud Storage ### Concept ![](https://1374779285-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-MFW3x2aqEO8GF2kr3VU%2Fuploads%2FTFX2Se1RUkr7WTCFvkEM%2Fimage.png?alt=media\&token=1d43cd12-5536-4b91-acf9-7ae09f98d052) ![](https://1374779285-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-MFW3x2aqEO8GF2kr3VU%2Fuploads%2FlniTsKAFflvFojBrqZMA%2Fimage.png?alt=media\&token=989148e2-286a-4c70-a03a-bef69804a3f6) * Here is the types of storage class ![](https://1374779285-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-MFW3x2aqEO8GF2kr3VU%2Fuploads%2F7FpHyBohPhIoJGToJkpX%2Fimage.png?alt=media\&token=33eb2e5d-8f71-4661-a3c9-4a907a509a07) ![](https://1374779285-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-MFW3x2aqEO8GF2kr3VU%2Fuploads%2FVozKpXWPFnmqogPLb5d7%2Fimage.png?alt=media\&token=b53f1989-baca-48ec-9311-90bc6a9d0d33) * An ACL is a mechanisms used to define who has access to your buckets and objects, as well as what the level of access to have. * The maximum number of ACL entries you can create for a bucket or object is 100. * The scope defines who , and permission defines the operation that they can do ![](https://1374779285-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-MFW3x2aqEO8GF2kr3VU%2Fuploads%2FNfSqONIh4SDVwTdQ4Z42%2Fimage.png?alt=media\&token=eb2455ae-d3a4-4381-8723-4dff74ac0c41) * Using signed URLs for granting limited time access tokens that can be used by any user instead of using account based authentication for controlling resource access. * You create a URL, the grants read or write access to a specific cloud storage resource, and specifies when this access expires. In Cloud Storage, objects are immutable, which means that an uploaded object cannot change throughout its storage lifetime. * When Object Versioning is enabled, you can list archived versions of an object, restore the live version of an object to an older state, or permanently delete an archived version as needed. * You can assign a lifecycle management configuration to a bucket. The configuration is a set of rules that apply to all objects in the buckets. When an object meets the criteria of one of the rules, Cloud Storage automatically performs a specified action on the object. E.g: downgrade the storage class of objects older than a year to Cloud line storage ![](https://1374779285-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-MFW3x2aqEO8GF2kr3VU%2Fuploads%2FaOKA6a7yU7JrUjhnCArq%2Fimage.png?alt=media\&token=03d4e0c7-99b7-4be9-aff5-371ab5a46fbd) * Transfer Appliance is a hardware appliance you can use to securely migrate large volumes of data from hundreds of terabytes up to one petabyte to Google Cloud without disrupting business operations. * The Storage Transfer Service enables high-performance imports of online data.The data source can be another Cloud Storage bucket, an Amazon S3 bucket, or an HTTP, HTTPS location. * Offline media import is a third-party service, where physical media such as storage arrays, hard disk drives, tapes, and USB flash drives is sent to a provider who uploads the data. ### Operation * Create a bucket ``` gsutil mb -p PROJECT_ID gs://BUCKET_NAME gsutil mb -p PROJECT_ID -c STORAGE_CLASS -l BUCKET_LOCATION -b on gs://BUCKET_NAME ``` * List and delete the buckets ``` gsutil ls -p PROJECT_ID gsutil rm -r gs://BUCKET_NAME ``` * Upload object to bucket ``` // Upload single object gsutil cp OBJECT_LOCATION gs://DESTINATION_BUCKET_NAME/ // Download object gsutil cp gs://BUCKET_NAME/OBJECT_NAME SAVE_TO_LOCATION ``` * Transfer bucket through api ``` POST https://storagetransfer.googleapis.com/v1/transferJobs { "description": "YOUR DESCRIPTION", "status": "ENABLED", "projectId": "PROJECT_ID", "schedule": { "scheduleStartDate": { "day": 1, "month": 1, "year": 2015 }, "startTimeOfDay": { "hours": 1, "minutes": 1 } }, "transferSpec": { "gcsDataSource": { "bucketName": "GCS_SOURCE_NAME" }, "gcsDataSink": { "bucketName": "GCS_SINK_NAME" }, "transferOptions": { "deleteObjectsFromSourceAfterTransfer": true } } } ``` ## Cloud SQL ### Concept * It is fully managed database service for MYSQL, Postgre SQL * Cloud SQL delivers high performance and scalability with up to 64 terabytes of storage capacity, 60,000 IOPS, and 624 gigabytes of RAM per instance. * Through synchronous replication to each zone's persistent disk, all writes made to the primary instance are replicated to disks in both zones before the transaction is reported as committed.In the event of an instance or zone failure, the persistent disk is attached to the standby instance and it becomes a new primary instance. * Cloud SQL also provides automated and on-demand backups with point-in-time recovery. * Allow import and export databases using MySQL dump or import and export CSV files. * Cloud SQL can also scale up, which does require machine restart or scale-out using read replicas. * Here is the connecting diagram of Cloud SQL ![](https://1374779285-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-MFW3x2aqEO8GF2kr3VU%2Fuploads%2FsFdCQbryOyVu0r8TW90k%2Fimage.png?alt=media\&token=30dc1edc-f54e-4f8a-a4e2-3e56232a3afc) * Here is the diagram of choosing Cloud SQL instead of using vm ![](https://1374779285-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-MFW3x2aqEO8GF2kr3VU%2Fuploads%2FwLon0hSFyUr9Kh1wm5VY%2Fimage.png?alt=media\&token=9cc21dd4-9319-4761-a486-80ff5ab63d97) ### Operation * Create instance ``` gcloud sql instances create INSTANCE_NAME \ --cpu=NUMBER_CPUS \ --memory=MEMORY_SIZE \ --region=REGION // Set the admin password gcloud sql users set-password root \ --host=% \ --instance INSTANCE_NAME \ --password PASSWORD // Start the instance gcloud sql instances patch INSTANCE_NAME \ --activation-policy=ALWAYS // Stop the instance gcloud sql instances patch INSTANCE_NAME \ --activation-policy=NEVER // Delete the instance gcloud sql instances delete INSTANCE_NAME ``` * Connect to SQL instance ``` // From cloud shell gcloud sql connect INSTANCE_ID --user=root // From local machine or vm mysql --host=INSTANCE_IP --user=root --password ``` * Export Data ``` gcloud sql export csv INSTANCE_NAME gs://BUCKET_NAME/FILE_NAME \ --database=DATABASE_NAME \ --offload \ --query=SELECT_QUERY ``` ## Cloud Spanner * Offers high availability, horizontal scalability and configurable replication * The database placement is configurable, meaning you can choose which region to put your database in. The replication of data will be synchronized across zones using Google's global fiber network. * Provide petabytes of capacity ![](https://1374779285-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-MFW3x2aqEO8GF2kr3VU%2Fuploads%2FncykbkLCaxvPuLzJKt04%2Fimage.png?alt=media\&token=94c8cc67-f36e-4836-b8d4-6936134a35d8) ![](https://1374779285-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-MFW3x2aqEO8GF2kr3VU%2Fuploads%2FvEoUHh5TDAt5I1urevhM%2Fimage.png?alt=media\&token=1cafa1df-a909-42ff-bb0d-8af92711c378) ## Cloud Firestore ### Concept * It is a fast, fully managed, server less, cloud native, NoSQL, document database * Its client libraries provide live synchronization and offline support and it's security features and integrations with firebase and GCP accelerate building truly server less apps. * It supports acid transactions so if any of the operations in the transaction fail and cannot be retried, the whole transaction will fail. * Also with automatic multi region replication and strong consistency, your data is safe and available even when disasters strike. ![](https://1374779285-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-MFW3x2aqEO8GF2kr3VU%2Fuploads%2FY28KMEG8CBmprYSG0TEk%2Fimage.png?alt=media\&token=bbf1426b-e05b-4d53-b193-982f28ab042e) * Datastore Mode: operate in Datastore mode, making it backwards compatible with Cloud Datastore, you can access Cloud Firestores, improve storage layer while keeping Cloud Datastore system behavior, Transactions are no longer limited to 25 entity groups, rights to an entity group are no longer limited to 1 per second * Native Mode: access all of the new Cloud Firestore features ![](https://1374779285-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-MFW3x2aqEO8GF2kr3VU%2Fuploads%2Fg2hYQAtPZptLh1TJ9eiO%2Fimage.png?alt=media\&token=8b72c235-4217-4bb4-bcc1-3002e8c9e0f2) ### Operation * Export Data to Cloud storage ``` gcloud firestore export gs://[BUCKET_NAME] ``` ## Cloud Bigtable * It is a fully managed NoSQL database with petabytes-scale and very low latency. * Ideal for Ad Tech, finTech and IoT * easy integration with open source big data tools ![](https://1374779285-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-MFW3x2aqEO8GF2kr3VU%2Fuploads%2FgIWzGg9i1UrjiH0EVgYJ%2Fimage.png?alt=media\&token=d9565d14-354b-4468-bc49-e7dce5c0f0fc) * Stores data in massively scalable tables, each of which is a sorted key value map. ![](https://1374779285-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-MFW3x2aqEO8GF2kr3VU%2Fuploads%2FU09ckGsyqTFdoYy5GS6Y%2Fimage.png?alt=media\&token=aee5f395-f086-4351-8da8-d6911e35c183) --- # Agent Instructions: Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter: ``` GET https://petercheng7788.gitbook.io/developer-note/devop/cloud/google-cloud/storage.md?ask= ``` The question should be specific, self-contained, and written in natural language. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.