Event History walkthrough with the .NET SDK
In order to understand how Workflow Replay works, this page will go through the following walkthroughs:
- How Workflow Code Maps to Commands
- How Workflow Commands Map to Events
- How History Replay Provides Durable Execution
- Example of a Non-Deterministic Workflow
How Workflow Code Maps to Commands
This walkthrough will cover how the Workflow code maps to Commands that get sent to the Temporal Service, letting the Temporal Service know what to do.
Step through the Workflow Definition below. Each step highlights the statements the Worker is running and shows the Commands it has issued so far.
1[Workflow]2public class PizzaWorkflow3{4 [WorkflowRun]5 public async Task<OrderConfirmation> RunAsync(PizzaOrder order)6 {7 // Activity Options omitted for brevity8 var totalPrice = order.Items.Sum(pizza => pizza.Price);910 var distance = await Workflow.ExecuteActivityAsync(11 (Activities act) => act.GetDistanceAsync(order.Address),12 options);1314 if (order.IsDelivery && distance.Kilometers > 25)15 {16 throw new ApplicationFailureException("Customer too far away for delivery");17 }1819 await Workflow.DelayAsync(TimeSpan.FromMinutes(30));2021 var bill = new Bill(22 CustomerId: order.Customer.CustomerId,23 OrderNumber: order.OrderNumber,24 Description: "Pizza",25 Amount: totalPrice);2627 var confirmation = await Workflow.ExecuteActivityAsync(28 (Activities act) => act.SendBillAsync(bill),29 options);3031 return confirmation;32 }33}
A basic Workflow Definition
This Workflow Definition takes a pizza order and does the work listed here. Step through it to see which statements the Worker handles on its own and which ones send a Command to the Temporal Service.
- Calculates the total price of the pizzas
- Determines the distance to the customer
- Fails if the customer is too far away for delivery
- Sleeps for 30 minutes
- Populates a record with billing information
- Sends a bill to the customer
Nothing yet.
Four statements in this Workflow Definition produce a Command:
| Statement | Command |
|---|---|
Workflow.ExecuteActivityAsync(GetDistanceAsync) | ScheduleActivityTask |
Workflow.DelayAsync(TimeSpan.FromMinutes(30)) | StartTimer |
Workflow.ExecuteActivityAsync(SendBillAsync) | ScheduleActivityTask |
return confirmation; | CompleteWorkflowExecution |
Everything else is an internal step. Totaling the order price, evaluating the distance, and populating the bill record all run in the Worker without contacting the Temporal Service.
How Workflow Commands Map to Events
The Commands that are sent to the Temporal Service are then turned into Events, which build up the Event History. The Event History is a detailed log of Events that occur during the lifecycle of a Workflow Execution, such as the execution of Workflow Tasks or Activity Tasks. Event Histories are persisted to the database used by the Temporal Service, so they're durable, and will even survive a crash of the Temporal Service itself.
These Events are what are used to recreate a Workflow Execution's state in the case of failure.
Step through the same Workflow Definition to see each Command the Worker issues and the Events the Temporal Service records in response.
1[Workflow]2public class PizzaWorkflow3{4 [WorkflowRun]5 public async Task<OrderConfirmation> RunAsync(PizzaOrder order)6 {7 // Activity Options omitted for brevity8 var totalPrice = order.Items.Sum(pizza => pizza.Price);910 var distance = await Workflow.ExecuteActivityAsync(11 (Activities act) => act.GetDistanceAsync(order.Address),12 options);1314 if (order.IsDelivery && distance.Kilometers > 25)15 {16 throw new ApplicationFailureException("Customer too far away for delivery");17 }1819 await Workflow.DelayAsync(TimeSpan.FromMinutes(30));2021 var bill = new Bill(22 CustomerId: order.Customer.CustomerId,23 OrderNumber: order.OrderNumber,24 Description: "Pizza",25 Amount: totalPrice);2627 var confirmation = await Workflow.ExecuteActivityAsync(28 (Activities act) => act.SendBillAsync(bill),29 options);3031 return confirmation;32 }33}
Commands and the Events they produce
This walkthrough keeps a running list of the Commands the Worker issues and the Events the Temporal Service records in response. Blue Events are the direct result of a Command. Pink Events are an indirect result.
Nothing yet.
Nothing yet.
Blue Events are the direct result of a Command. Pink Events are an indirect result, such as the Events the Temporal Service records when a Worker starts or finishes a Task:
| Command | Direct Event | Indirect Events |
|---|---|---|
ScheduleActivityTask | ActivityTaskScheduled | ActivityTaskStarted, ActivityTaskCompleted |
StartTimer | TimerStarted | TimerFired |
How History Replay Provides Durable Execution
Now that you have seen how code maps to Commands, and how Commands map to Events, this next walkthrough will take a look at how Temporal uses Replay with the Events to provide Durable Execution and restore a Workflow Execution in the case of a failure.
This code walkthrough will begin by walking through a Workflow Execution, describing how the code maps to Commands and Events. There will then be a Worker crash halfway through, explaining how Temporal uses Replay to recover the state of the Workflow Execution, ultimately resulting in a completed execution that's identical to one that had not crashed.
1[Workflow]2public class PizzaWorkflow3{4 [WorkflowRun]5 public async Task<OrderConfirmation> RunAsync(PizzaOrder order)6 {7 // Activity Options omitted for brevity8 var totalPrice = order.Items.Sum(pizza => pizza.Price);910 var distance = await Workflow.ExecuteActivityAsync(11 (Activities act) => act.GetDistanceAsync(order.Address),12 options);1314 if (order.IsDelivery && distance.Kilometers > 25)15 {16 throw new ApplicationFailureException("Customer too far away for delivery");17 }1819 await Workflow.DelayAsync(TimeSpan.FromMinutes(30));2021 var bill = new Bill(22 CustomerId: order.Customer.CustomerId,23 OrderNumber: order.OrderNumber,24 Description: "Pizza",25 Amount: totalPrice);2627 var confirmation = await Workflow.ExecuteActivityAsync(28 (Activities act) => act.SendBillAsync(bill),29 options);3031 return confirmation;32 }33}
A Client requests the Workflow Execution
The walkthrough begins with a request to execute this Workflow Definition with input data about the customer and the pizzas ordered. The Temporal Service records WorkflowExecutionStarted, always the first Event of a Workflow Execution, and that Event holds the input data.
Nothing yet.
- WorkflowExecutionStarted(customer, pizzas ordered)
The walkthrough covers four phases:
- Original execution: the Client starts the Workflow Execution, and the Commands the Worker issues become Events in the Event History.
- Worker crash: the Worker dies partway through a Workflow Task. When the Workflow Task Timeout elapses, 10 seconds
by default, the Temporal Service records
WorkflowTaskTimedOutand schedules a new Workflow Task. - History Replay: a Worker requests the Event History and re-executes the Workflow code with the original input,
which the
WorkflowExecutionStartedEvent stores. Commands the Worker creates during Replay are matched against the Event History instead of being issued, so Activities don't run again. The Worker uses the results stored in theActivityTaskCompletedEvents. - Execution resumes: past the point of the crash, the Event History holds no matching Events, so the Worker issues Commands for real again until the Workflow Execution completes. The result is identical to an execution that never crashed.
Example of a Non-Deterministic Workflow
Now that Replay has been covered, this section will explain why Workflows need to be deterministic in order for Replay to work.
A Workflow is deterministic if every execution of its Workflow Definition produces the same Commands in the same sequence given the same input.
As mentioned in the How History Replay Provides Durable Execution
walkthrough, in the case of a failure, a Worker requests the Event History to replay it. During Replay, the Worker runs
the Workflow code again to produce a set of Commands which is compared against the sequence of Commands in the Event
History. When there’s a mismatch between the expected sequence of Commands the Worker expects based on the Event History
and the actual sequence produced during Replay (due to non-determinism), Replay will be unable to continue.
To better understand why Workflows need to be deterministic, it's helpful to look at a Workflow Definition that violates it. In this case, this code will walk through a Workflow Definition that breaks the determinism constraint with a random number generator.
1[Workflow]2public class GenerateDailyReport3{4 private static readonly Random random = new Random();56 [WorkflowRun]7 public async Task<string> RunAsync()8 {9 // Activity Options and logger declaration omitted for brevity10 var salesData = await Workflow.ExecuteActivityAsync(11 (Activities act) => act.ImportSalesDataAsync(),12 options);1314 if (random.Next(100) >= 50)15 {16 await Workflow.DelayAsync(TimeSpan.FromHours(4));17 }1819 Logger.LogInformation("Preparing to run daily report");2021 return await Workflow.ExecuteActivityAsync(22 (Activities act) => act.RunDailyReportAsync(),23 options);24 }25}
The ImportSalesData Activity runs
As this Workflow executes step by step, the first line that results in a Command is the call to the ImportSalesData Activity. The Activity Execution succeeds, so the Temporal Service logs three Events to the Event History.
- ScheduleActivityTask(ImportSalesData)
- ActivityTaskScheduled(ImportSalesData)
- ActivityTaskStarted
- ActivityTaskCompleted
During the first execution, the random number is 84, so the Workflow starts a Timer and the Event History records
TimerStarted and TimerFired. During Replay, the random number is 14, so the Workflow skips the Timer and produces a
ScheduleActivityTask Command where the Event History expects StartTimer. That mismatch is what makes Replay fail.
Note that non-deterministic failures do not fail the Workflow Execution by default. A non-deterministic failure is considered a Workflow Task Failure which is considered a transient failure, meaning it retries over and over. Users can also fix the source of non-determinism, perhaps by removing the Activity, and then restart the Workers. This means that this type of failure can recover by itself. You can also use a strategy called versioning to address this non-determinism error. See versioning to learn more.
For more information on how Temporal handles Durable Execution or to see these walkthroughs in video format with more explanation, check out our free, self-paced courses: Temporal 102 and Versioning Workflows.
Temporal Applications Support Non-Deterministic Operations
We want to emphasize that although your Workflows themselves need to be deterministic, your application itself does not!
Remember that pretty much anything that interacts with the external world is inherently non-deterministic:
- Calling LLM APIs
- Querying databases
- Reading or writing files
- Making HTTP requests to external services
Good news: Your Temporal application can absolutely handle all of these operations. While your Workflow must be deterministic, your application absolutely can handle any type of non-deterministic operation, including those listed above. This gives you the best of both worlds—the crash-proof reliability of a Workflow and the resiliency of Activities which have built-in support for retries.