Day #15 with Cloud Workflows: built-in cloud logging function

In the two previous episodes, we saw how to create and call subworkflows, and we applied this technique to making a reusable routine for logging with Cloud Logging. However, there’s already a built-in function for that purpose! So let’s have a look at this integration.


To call the built-in logging function, just create a new step, and make a call to the sys.log function:


- logString:
    call: sys.log
    args:
        text: Hello Cloud Logging!
        severity: INFO


This function takes a mandatory parameter: text. And an optional one: severity.


The text parameter accepts all types of supported values, so it’s not only string, but all kinds of numbers, as well as arrays and dictionaries. Their string representation will be used as text.


The optional severity parameter is an enum that can take the values: DEFAULT, DEBUG, INFO, NOTICE, WARNING, ERROR, CRITICAL, ALERT, EMERGENCY, with DEFAULT being… the default value if you don’t specify a severity!


Here’s another example with a dictionary as parameter, which will be output as text in the logs, and a severity of WARNING:


- createDict:
    assign:
        - person:
            name: Guillaume
            kids: 2
- logDict:
    call: sys.log
    args:
        text: ${person}
        severity: WARNING


Looking at the results in the cloud logging console, you will see both messages appear:



Don’t hesitate to have a look at reference documentation to find more about the available built-in functions.


Day #14 with Cloud Workflows: Subworkflows

Workflows are made of sequences of steps and branches. Sometimes, some particular sequence of steps can be repeated, and it would be a good idea to avoid error-prone repetitions in your workflow definition (in particular if you change in one place, and forget to change in another place). You can modularize your definition by creating subworkflows, a bit like subroutines or functions in programming languages. For example, yesterday, we had a look at how to log to Cloud Logging: if you want to log in several places in your workflow, you can extract that routine in a subworkflow.


Let’s see that in action in the video below, and you can read all the explanations afterwards:



First things first, let’s step back and look at the structure of workflow definitions. You write a series of steps, directly in the main YAML file. You can move back and forth between steps thanks to jumps, but it wouldn’t be convenient to use jumps to emulate subroutines (remember the good old days of BASIC and its gotos?). Instead, Cloud Workflows allows you to separate steps under a “main”, and subroutines under their own subroutine name.


So far we had just a sequence of steps:


- stepOne:
    ...
- stepTwo:
    ...
- stepThree:
  ...


Those steps are implicitly under a main routine. And here’s how to show this main routine explicitly, by having that main block, and steps underneath:


main:
    steps:
        - stepOne:
            ...
        - stepTwo:
            ...
        - stepThree:
          ...


To create a subworkflow, we follow the same structure, but with a different name than name, but you can also pass parameters like so:


subWorkflow:
    params: [param1, param2, param3: "default value"]
    steps:
        - stepOne:
            ...
        - stepTwo:
            ...
        - stepThree:
          ...


Notice that you can pass several parameters, and that parameters can have default values when that parameter is not provided at the call site.


Then in your main flow, you can call that subworkflow with a call instruction. Let’s take a look at a concrete example, that simply concatenates two strings:


main:
    steps:
        - greet:
            call: greet
            args:
                greeting: "Hello"
                name: "Guillaume"
            result: concatenation
        - returning:
            return: ${concatenation}

greet:
    params: [greeting, name: "World"]
    steps:
        - append:
            return: ${greeting + ", " + name + "!"}


In the call instruction, we pass the greeting and name arguments, and the result will contain the output of the subworkflow call. In the subworkflow, we defined our parameters, and we have a single step just return an expression which is the desired greeting message concatenation.


One last example, but perhaps more useful than concatenating strings! Let’s turn yesterday’s Cloud Logging integration into a reusable subworkflow. That way, you’ll be able to call the log subworkflow as many times as needed in your main workflow definition, without repeating yourself:


main:
  steps:
    - first_log_msg:
        call: logMessage
        args:
          msg: "First message"
    - second_log_msg:
        call: logMessage
        args:
          msg: "Second message"
   
logMessage:
  params: [msg]
  steps:
    - log:
        call: http.post
        args:
            url: https://logging.googleapis.com/v2/entries:write
            auth:
                type: OAuth2
            body:
                entries:
                    - logName: ${"projects/" + sys.get_env("GOOGLE_CLOUD_PROJECT_ID") + "/logs/workflow_logger"}
                      resource:
                        type: "audited_resource"
                        labels: {}
                      textPayload: ${msg}


And voila! We called our logMessage subworkflow twice in our main workflow, just passing the text message to log into Cloud Logging.

Day #13 with Cloud Workflows: Logging with Cloud Logging

Time to come back to our series on Cloud Workflows. Sometimes, for debugging purposes or for auditing, it is useful to be able to log some information via Cloud Logging. As we saw last month, you can call HTTP endpoints from your workflow. We can actually use Cloud Logging’s REST API to log such messages! Let’s see that in action.


- log:
    call: http.post
    args:
        url: https://logging.googleapis.com/v2/entries:write
        auth:
            type: OAuth2
        body:
            entries:
                - logName: ${"projects/" + sys.get_env("GOOGLE_CLOUD_PROJECT_ID") + "/logs/workflow_logger"}
                  resource:
                    type: "audited_resource"
                    labels: {}
                  textPayload: Hello World from Cloud Workflows!


We call the https://logging.googleapis.com/v2/entries:write API endpoint to write new logging entries. We authenticate via OAuth2—as long as the service account used for the workflow execution allows it to use the logging API. Then we pass a JSON structure as the body of the call, indicating the name of the logger to use, which resources it applies to, and also the textPayload containing our text message. You could also use a ${} expression to log more complex values.


Once this workflow definition is done and deployed, you can execute it, and you should see in the logs your message appear:



Voila! You can log messages to Cloud Logging!


Let's recap in this video:



In the next episode, we’ll take advantage of subworkflows, to create a reusable set of steps that you will be able to call several times throughout your workflow definition, without repeating yourself, by turning this logging example into a subworkflow.


Day #12 with Cloud Workflows: loops and iterations

In previous episodes of this Cloud Workflows series, we’ve learned about variable assignment, data structures like arrays, jumps and switch conditions to move between steps, and expressions to do some computations, including potentially some built-in functions. 


With all these previous learnings, we are now equipped with all the tools to let us create loops and iterations, like for example, iterating over the element of an array, perhaps to call an API several times but with different arguments. So let’s see how to create such an iteration!




First of all, let’s prepare some variable assignments:


- define:
    assign:
        - array: ['Google', 'Cloud', 'Workflows']
        - result: ""
        - i: 0


  • The array variable will hold the values we’ll be iterating over.

  • The result variable contains a string to which we’ll append each values from the array.

  • And the i variable is an index, to know our position in the array.


Next, like in a for loop of programming languages, we need to prepare a condition for the loop to finish. We’ll do that in a dedicated step:


- checkCondition:
    switch:
        - condition: ${i < len(array)}
          next: iterate
    next: returnResult


We define a switch, with a condition expression that compares the current index position with the length of the array, using the built-in len() function. If the condition is true, we’ll go to an iterate step. If it’s false, we’ll go to the ending step (called returnResult here).


Let’s tackle the iteration body itself. Here, it’s quite simple, as we’re just assigning new values to the variables: we append the i-th element of the array into the result variable, and we increment the index by one. Then we go back to the checkCondition step.


- iterate:
    assign:
        - result: ${result + array[i] + " "}
        - i: ${i+1}
    next: checkCondition


Note that if we were doing something more convoluted, for example calling an HTTP endpoint with the element of the array as argument, we would need two steps: one for the actual HTTP endpoint call, and one for incrementing the index value. However in the example above, we’re only assigning variables, so we did the whole body of the iteration in this simple assignment step.


When going through the checkCondition step, if the condition is not met (ie. we’ve reached the end of the array), then we’re redirected to the returnResult step:


- returnResult:
    return: ${result}


This final step simply returns the value of the result variable.


Days #11 with Cloud Workflows: sleeping in a workflow

Workflows are not necessarily instantaneous, and executions can span over a long period of time. Some steps may potentially launch asynchronous operations, which might take seconds or minutes to finish, but you are not notified when the process is over. So when you want for something to finish, for example before polling again to check the status of the async operation, you can introduce a sleep operation in your workflows.



To introduce a sleep operation, add a step in the workflow with a call to the built-in sleep operation:


- someSleep:
    call: sys.sleep
    args:
        seconds: 10
- returnOutput:
    return: We waited for 10 seconds!


A sleep operation takes a seconds argument, where you can specify the number of seconds to wait.


By combining conditional jumps and sleep operations, you can easily implement polling some resource or API at a regular interval, to double check that it completed.


 
© 2012 Guillaume Laforge | The views and opinions expressed here are mine and don't reflect the ones from my employer.