Most applications at the enterprise, mid-size, or even startup levels, would have one or more 3rd party APIs they integrate with. It may start off with just one component talking to the external API, but can very quickly grow into multiple systems talking to that 3rd party API. It’s not a good practice to have direct integrations for all those different systems. Instead, a good pattern is to have a 3rd Party API Integration Service, deployed as a company wide API, where this API acts as the gateway between the 3rd party system and systems within your application / enterprise.
The benefits of this pattern are mainly that the integration service insulates the rest of the ecosystem from the external API. This helps avoid negative coupling spread throughout the system as well as redundant codebases that could all very quickly go out of sync. All interactions are done in one system and one codebase, which makes it significantly simpler to maintain. Most such integration services often serve multiple functions.
Authorization – All of the auth code to the 3rd party API is managed in this one service.
Caching – If the 3rd party API is expensive or not as performant as you’d like it to be, you can add a fast caching layer here (and use something like Redis for recent repeat calls).
Schema Transform – You can transform the schema to a more company compliant standard.
Data Filtering – This would be a simple place to filter out certain results or certain fields within results.
Model Enrichment – You can enrich the 3d party response by adding internal system identifiers, which will make it far easier for downstream callers to resolve entities.
Local DB persistence – All fetched data can be persisted locally, meaning that in a future scenario where the 3rd party API may not be available to integrate with, you still have the original data.
And the fact that you have this integration service means that you can switch 3rd party services without the internal systems having to make any changes. A new version of the integration service will abstract away the new external API and convert the data to the same format expected by the downstream callers.
For very basic Web APIs, HTTP triggered Azure functions are a very efficient and cost effective way to implement the service. One inconvenience though is that by default it does not come hooked up with an appsettings.json file or an IConfiguration instance. Fortunately, it’s not very difficult to hook those up.
Step 1
Add the following NuGet package to your project.
Microsoft.Azure.Functions.Extensions
Step 2
Add a startup.cs file to your project and insert the following code.
[assembly: FunctionsStartup(typeof(Startup))]
public class Startup : FunctionsStartup
{
public override void ConfigureAppConfiguration(
IFunctionsConfigurationBuilder builder)
{
var context = builder.GetContext();
builder.ConfigurationBuilder
.AddJsonFile(
Path.Combine(
context.ApplicationRootPath,
"appsettings.json"),
optional: true,
reloadOnChange: false)
.AddJsonFile(
Path.Combine(
context.ApplicationRootPath,
$"appsettings.{context.EnvironmentName}.json"),
optional: true,
reloadOnChange: false)
.AddEnvironmentVariables();
}
}
Step 3
Add your appsettings.json file and setup any configuration properties you need in there.
{
"YourProperty": "some-value-here"
}
Step 4
Inject IConfiguration into your constructor. Note that the default function template won’t have a constructor as it uses a static class. Remove the static, and add the constructor.
For some time, I had been generating my Git repositories using Visual Studio, which automatically generated a “master” branch. Subsequently, I would create a “main” branch, push it, and following that, delete the “master” branch. All this time, I mistakenly believed that this behavior was determined by Visual Studio. However, I recently learned that this is actually a global Git configuration setting, which can be updated with a single command.
git config --global init.defaultBranch main
Well, this will save me 3-4 minutes every time I initialize a new repository.
Okay, so this may not be the most efficient code – so you may not want to use this in a context where milli seconds matter. But say you have a model class (returned by another library or from an API) and you need to extend it by adding a single property to it. The simplest way to do this is to derive a new class, add the property to it and then to add a constructor that copy-constructs from base to derived. Bit of a hassle having to copy every single property though, although I’d imagine some of those new AI powered IDEs could auto-generate code for you. Even then, if the base class ever changes, you will need to update this constructor and if you forget to do that, there’s a risk of random buggy behavior. So, what’s the hack? Use JsonConvert to serialize the base object, then deserialize it to derived and then populate the new field or fields that’s been added to derived.
static void Main()
{
ModelEx model = Foo(new Model
{
Id = 1000,
Name = "Mary Miller",
Description = "Senior Accountant"
});
model.Enabled = true;
}
private static ModelEx Foo(Model model)
{
var json = JsonConvert.SerializeObject(model);
return JsonConvert.DeserializeObject<ModelEx>(json);
}
class ModelEx : Model
{
public bool Enabled { get; set; }
}
class Model
{
public int Id { get; set; }
public string Name { get; set; }
public string Description { get; set; }
}
If you need your WebJob to run a task and exit, or if you need to conditionally exit it under certain scenarios, and you thought it’s just a matter of exiting the worker service’s ExecuteAsync method, you’d be wrong. Even after your worker code has stopped running, the parent host stays alive. The right way to exit your WebJob is to inject IHostApplicationLifetime into your worker, and then the last line of code in your ExecuteAsync would be a call to its StopApplication method.
public Worker(ILogger<Worker> logger, IHostApplicationLifetime appLifeTime)
{
_logger = logger;
_appLifetime = appLifeTime;
}
protected override async Task ExecuteAsync(CancellationToken stoppingToken)
{
int runs = 0;
while (!stoppingToken.IsCancellationRequested && runs++ < 3)
{
// do some task here
await Task.Delay(2000, stoppingToken);
}
_appLifetime.StopApplication();
}
With regular App Services, if you enabled Application Insights, your ILogger logs will get written into Application Insights without having to do anything special. This is not the case with WebJobs though. After wasting a few hours trying to figure out why my logs were not showing up, here’s a solution I found that worked.
Step 1
Install the latest version of Microsoft.Azure.WebJobs.Logging.ApplicationInsights.
And that’s it. Remember that there’s usually a 1-2 minute delay before your logs show up in App Insights, and it could be as high as 3-4 minutes if it’s the first time you deploy. So, don’t panic if you don’t see the logs right away. Happy debugging!
Disclaimer – Copyright Rocket Mortgage, LLC [2022 – 2023]. The author (Nish Nishant) has been authorized to post this article under license from Rocket Mortgage. This article is is the property of Rocket Mortgage, LLC and is not authorized for redistribution.
Author’s Note – I wrote this article for my employer’s public facing technology blog, but for various reasons, that blog was retired. Since I did not want to lose the content forever, I am re-publishing it here with explicit approval, and hence the disclaimer above.
Abstract
As software systems grow in number and complexity, they end up being inadvertently dependent on each other. This slows down the delivery velocity across technology companies. Teams are forced to delay deliverables as they wait for other cohorts to finish functionality that they’re reliant on. This delay also results in teams not evolving their legacy components.
Solutions to these challenges include defining capability domains, creating boundary APIs and using an enterprise-wide event stream to achieve clean separation across capability domains. This article focuses on the core patterns required to achieve this solution and does not recommend any particular tool-stacks other than as coincidental examples.
Capability Domains
Defining and mapping a technology enterprise’s software systems into domains based on capability is a common first step in avoiding dependencies between systems. Typically, these domains match the technology enterprise’s organizational structure, though it isn’t required. Each capability domain owns the functionality to implement its specific capability, and ideally, there wouldn’t be any overlap across domains.
Increasing the autonomy of the domain is one of the goals of capability domains, which means that systems owned by the domain are considered internal to each domain. As a result, the teams building those systems have the freedom to dictate their technology roadmaps.
A common anti-pattern here is when internal systems integrate directly with other systems from another domain. Doing this is a form of negative coupling that negates the advantages of defining and mapping enterprise-wide capability domains. The preferred way to expose a domain’s functionality to another with minimal coupling is through domain boundaries.
Domain Boundaries
Domain boundaries are public interfaces on the edge of the domain. Typically, these public interfaces are implemented as public REST APIs with published service contracts. Domains within an enterprise need to set mutual expectations that they will only integrate with other domains via that domain’s boundary APIs. Doing so solves the problem of negative coupling that arises as a side effect of directly integrating with internal systems outside their domain.
The boundary APIs also need to be backwards compatible, and it’s good practice to maintain a formal API versioning system. Boundary APIs themselves lack the autonomy internal domain systems enjoy, but this is a very small price to avoid roadmap-hindering dependencies.
Enterprise Event Streams
An event stream is a continuous stream of events broken down into named event topics. An enterprise event stream is a shared event stream where all capability domains publish and receive messages.
Event streams are typically built on data streaming platforms such as Confluent Kafka, Amazon Kinesis or Azure Event Hubs. Regardless of the vendor you choose, they all have a standard, core set of features to stream event messages and are all built for high scalability, concurrency, resiliency and performance. However, more critical than your vendor choice is standardizing event schemas, capability owners and controlling the access to those events.
Combining Domains and Event Streams
Figure 1 – Domains sitting around an Enterprise Event Stream
Once implemented together, capability domains and an enterprise event stream can help an enterprise achieve a high degree of independence across domains. Figure 1 shows multiple domains (X, Y, Z and W); none of those have direct coupling to another domain. They only integrate through the enterprise event stream. There are also multiple consumers (A, B, C and D) and those consumers either integrate with the enterprise event stream or with the domain API, but never directly with an internal domain component such as the App APIs inside domain X.
There are two fundamental tenets to this pattern:
Domains directly integrate with other domains solely through domain boundary APIs (often referred to as domain APIs).
Domains produce and consume enterprise events via the enterprise event stream.
The enterprise events produced by a domain and their domain API are the only public contracts exposed. They’re the only aspects of their domain that must follow enterprise standards and be backwards compatible. Everything else that powers that domain’s capability is internal to the domain, which gives the domain owners a high level of freedom concerning their choice of implementation languages, libraries, container technologies, cloud resource preferences and software standards.
For example, one domain may use .NET and C# on Microsoft Azure because they have team members proficient in those technologies. Even if the rest of the enterprise uses Java on Amazon Web Services, both domains publish standardized events to the enterprise event stream and expose a public REST API, which are agnostic to their technology stack of choice.
Event Payload Patterns
Figure 2 – Common Payload Patterns
Different enterprises choose approaches that work best for their business requirements. In general, there are three types of payload patterns that are used for enterprise events. Figure 2 illustrates three different consumers: A, B and C, each of them using one of the payload patterns. Consumer A uses the Lightweight pattern, consumer B uses the Full Payload pattern, while Consumer C uses the Hybrid approach.
Lightweight Events
For lightweight events, the event payload is often a single identifier or a URI (uniform resource identifier). Consumers receive the event and then make an API call to a domain API to retrieve the current state of the data associated with that event. The benefits of lightweight events are that you save on data size as the events are extremely minimal and you do not need to worry about event ordering since the consuming systems always retrieve the latest state of the data from the source domain. The disadvantage of lightweight events is that the consumer must make an API call to retrieve the data it needs.
Full Payloads
In full payloads, the event payload is a complete snapshot of the state of the event data at a fixed point in time. Consuming systems have all the information they need when they receive the event. These systems save on having to make a call to a domain API. The disadvantage of full payloads is that the order in which the events are processed matters, which is a solved problem as all of the popular vendors support guaranteed message ordering.
Hybrid Approach
In the hybrid approach, the event payload is a partial dataset of the event state. The payload includes the source identifiers and URIs, and a subset of the state of the event. This allows some consumers to determine if they need to retrieve further data from the domain API while allowing other consumers to operate off the subset state of the data in the event payload. The hybrid approach is considered the best of both worlds by many.
There is no perfect way to do this, and it’s a sign of a mature enterprise if they choose to use more than one of these approaches to fit specific requirements.
Conclusion
Highly independent and fast-delivering technology companies use an enterprise event stream and cataloging of multiple capability domains. They couple this with domain owners who follow standards, and the company grows through the additive effects of the capability domains’ advancements.
Enumerations in C# are ints by default, which is fine most of the time. That said, there are situations where you’d want to store and retrieve them using their string names, especially when you are persisting them to a database. The string names make them more readable for one reason. For another, a re-ordering of the enums (caused inadvertently by the addition or removal of an entry) would change their int values, which can potentially break your data when reading them back from a database. Consider the following types – Model and Seasons.
enum Seasons
{
Spring,
Summer,
Fall,
Winter
}
class Model
{
public string Name { get; set; } = String.Empty;
public Seasons Season { get; set; }
}
If you serialize an instance of Model, you’d see the int value for the Season property.
var model = new Model
{
Name = "Test Model",
Season = Seasons.Fall,
};
var json = JsonConvert.SerializeObject(model, Formatting.Indented);
Console.WriteLine(json);
Output looks like this.
{
"Name": "Test Model",
"Season": 2
}
It’s fairly easy to change this behavior though. Just add a JsonConverter attribute to the Season property.
class Model
{
public string Name { get; set; } = String.Empty;
[JsonConverter(typeof(StringEnumConverter))]
public Seasons Season { get; set; }
}
Output is now nice and clean.
{
"Name": "Test Model",
"Season": "Fall"
}
You get the same behavior both when serializing and deserializing, and the latter is important when reading data back from a database.
How about if you have a collection of enums though? That’s simple too, just replace the JsonConverter attribute with the following JsonProperty attribute.
class Model
{
public string Name { get; set; } = String.Empty;
[JsonProperty(ItemConverterType = typeof(StringEnumConverter))]
public List<Seasons> Seasons { get; set; } = new List<Seasons>();
}
(click on the image to view it in full resolution)
This is an extension of the asynchronous endpoint pattern where the consumer can register a callback URL, which is invoked when the task is completed. This only works if the consumer has the option to expose an externally hittable REST endpoint. The advantage is that the consumer does not have to keep polling the service to see if the task has completed.
Once the task completes, the service invokes the callback URL. A common approach is to send just a task id to the callback, and the consumer then invokes a call to the service to fetch the full payload. A less common approach (if sensitive info is not included in the payload) is to send the entire result-set to the callback endpoint.
On the service side, the implementation is done by reacting to the task completed event by invoking the registered callback for the completed task. There are several ways to do this. A simple approach is for the task worker itself to invoke the callback. The disadvantage there is that it’s complex to now support retries if the callback endpoint is down. With AWS, a common approach is to update the data in a DynamoDB table and for a Lambda to be triggered off that, and the Lambda would then invoke the callback. A similar approach for Azure is to have a Function triggered off Table storage or Cosmos DB, which would invoke the callback. Both AWS and Azure have mechanisms to add retry to the function/lambda, to handle scenarios where the callback URI is down.
Nish Nishant's technology blog 