I wrote about using Google cloud trace and debugger before in 2020. I’ve not seen large adoption in big GCP customers but those are the best development experience that makes Google cloud special. I’m covering the improvements in the latest versions of the tools. OpenCensus is getting merged to OpenTelemetry. I experienced problems with OpenCensus `pip install` and would recommend using OpenTelemetry for Python3. I wills start with the obvious of Cloud Trace, then talk about how to instrument Cloud Debugger, then go easy on Cloud logging.
Cloud Trace
is the ultimate tool for finding performance bottleneck and understand which of the deployed microservices are taking time. A new feature I’m showing here is to allow tracing all Python3 Flask exposed endpoints without creating any TimeSpan. You can observe the trace from running locally and from running in another cloud. 1st step is to create a service account and grant it Cloud Trace Agent IAM role in the project. To make things easy, the example is in a single project. Follow the steps in the OpenTelemetry for python tutorial and see my commit to add Cloud Trace to a sample application.
Flask is instrumented in api.py. I followed the FlaskInstrumentor section. Observe the /promotions trace detail in the screenshot.
The start_span method to add trace for database operations is in query.py. Observe the child trace span of Firestore operation: get all promotions in the screenshot.
Cloud Debugger
allows debugging microservices in production without impacting performance. I followed the Setting Up Cloud Debugger for Python steps to enable it. Because I want to use Cloud Debugger locally and on Oracle cloud infrastructure, I had to pass the service account json file. The commit contains what code lines are added to enable it. Once you take a snapshot, you’d get something like this where the response dict has a key starting with pD containing an element of service-category: null.
Make sure the service account running the microservice has Cloud Debugger Agent IAM role and the user has Cloud Debugger User IAM role. The drop down list of Application contains the version. The best practice is to pass the version per git commit or docker image tag. Look for L35 in the cloud build file on how to pass VERSION as an environment variable. With CI CD, inject such VERSION environment variable to generate a Kubernetes deployment containing such variable. Pass the variable to the line that enables the debugger. os.environ.get(“VERSION”, str(datetime.now())) uses the current datetime if no such environment variable is found.
Cloud Logging
is the best way to write your application logs. I created a custom logging method in Python that writes to a specific log name. The log name shows in the format of logName: “projects/YOUR_PROJECT_ID/logs/LOG_NAME”. The 2nd parameter allows you to pass different log severity levels for further customization. Make sure the service account running your application has Logs Writer IAM role.
