Generating and sending task tokens
You can easily generate a task token by writing a couple of lines of ASL. Example 5-5 shows an example script.
Example 5-5. Step Functions ASL script to publish an event onto a custom event bus that will wait for the callback with a task token
{
“StartAt”: “Get finance clearance”,
“States”: {
“Finance clearance”: {
“Type”: “Task”,
“Resource”: “arn:aws:states:::events:putEvents.waitForTaskToken”,
“HeartbeatSeconds”: 6000,
“Parameters”: {
“Entries”: [
{
“Detail”: {
“metadata”: {
“version”: “1.0”,
“trace_id”: “djsf34hgsad8ndc2”,
“created_at”: “2023-12-21T11:10:54.000Z”,
“customer_accounts_request_id.$”: “$$.Task.Token”,
“domain”: “Customers”,
“service”: “customer-accounts”,
“category”: “operational_event”,
“type”: “task”,
“name”: “account_requested”
},
“data”: {
“customer_id.$”: “$.customer_id”,
“other”: “details go here”
}
},
“DetailType”: “event”,
“EventBusName”: “customers-event-bus”,
“Source”: “customer-account-process”
}
]
},
“Next”: “Another Step”
}
}
}
The HeartbeatSeconds value in Example 5-5 indicates the timeout period for this task. The task will fail, timing out, once this value is met. You can, however, request a heartbeat extension to have Step Functions reset the timer.
The two important attributes in this script are Resource and customer _accounts_request_id. waitForTaskToken instructs the Step Functions workflow to wait at this step until the task token generated by $$.Task.Token is returned.
You can attach the task token to any custom attribute of yours by assigning the token generated by $$.Task.Token. It’s a good practice to use an attribute that is part of the custom event schema that the consuming service understands.
Things to be aware of while using callbacks with a task token
Here are a few important points you must be familiar with when working with callback task tokens in Step Functions:
Task token carriers need not be EventBridge events.
Example 5-5 uses a custom EventBridge event to carry the task token to the consuming application as it demonstrates event choreography between microser‐ vices, but this is not required. You can wrap the task token in an SQS message and have it handled by a Lambda function, for example.
Be flexible on the custom event category that contains the task token.
Depending on how you classify your events, the custom event that contains the task token can be a domain event, or you may use an operational event category for this purpose.
You can use multiple task tokens in a workflow.
For brevity, our example used a single task token. You can use multiple task tokens at different parts of your workflow to pause and resume the different arms of the flow. Step Functions manages the tasks and their tokens.
Task providers (token-consuming targets) have a responsibility.
Though event consumers are viewed as being agnostic of producers, in dis‐ tributed orchestration, the consumers have an extra role to play. The primary orchestrator that sends the task token needs to know the consumers who fulfill each task. Only those consumers are expected to return the task tokens, and the primary orchestrator’s service will have special event filters for this purpose.
Avoid task timeouts by extending the heartbeat interval.
Step Functions provides three options when resubmitting task tokens:
- SendTaskSuccess resumes the workflow from the paused step.
- SendTaskFailure fails the execution.
- SendTaskHeartbeat makes the workflow step wait longer.
When a task needs to wait longer, it’s important that you extend the heartbeat interval to prevent it from timing out and making the workflow fail.
The code snippet in Example 5-6 demonstrates how you notify Step Functions of success.
Example 5-6. Sample code to send task success information along with the token to a Step Functions workflow
…
const taskToken = event.metadata.customer_accounts_request_id; const output = JSON.stringify(event);
// Check the task completion status in the event data
const params = {
output: output,
taskToken: taskToken
};
const result = await sfn.sendTaskSuccess(params).promise()
…
The Saga Pattern
A saga is a long story recounting a series of events. In software, the term is used to refer to a business process or workflow comprising a series of steps. Each step can be a self-contained task known as a transaction. A local transaction can itself have multiple steps/operations, but as a whole, all of those steps/operations must succeed—the transaction is atomic.
Enterprise business processes can be complex, involving the services of several inter‐ nal and external applications, and often human interactions as well. In most cases, if a step (i.e., a transaction) fails, the preceding steps can be reverted through a series of compensating transactions that bring them back to their initial state. If the saga andits steps are not time-critical, it may be possible to retry a step that fails, allowing it (and the saga as a whole) to succeed. The circuit breaker pattern discussed earlier in this chapter is common in those situations. An additional complication is that some critical business processes go through different stages and mark some of the steps or transactions as pivot transactions. In this case, successful completion of a pivot transaction might disallow reversal of the whole saga; once a certain point has been reached, there’s no other option but to carry on until the entire saga is complete.
In distributed transaction scenarios, the saga pattern provides a useful way to maintain consistency across microservices. Both choreography and orchestration are popular implementation choices. As you learned in the previous sections, in choreography each step or transaction is performed by a microservice, and it emits an event that becomes the indicator for progressing, retrying, or rolling back the saga. With orchestration, the orchestrator (i.e., the Step Functions workflow) controls the series of steps and will have the necessary paths to handle the failures and retries.
Many business use cases of the saga pattern will use both approaches. While archi‐ tecting, you must study the business requirements thoroughly to determine whether choreography, orchestration, or a combination of the two suits you better.