Reliability and Resiliency for Cloud Connected Applications

Building cloud connected applications that are reliable is hard. At the heart of building such a system is a solid architecture and focus on resiliency. We're going to explore what that means in this post.

When I first started development on FastBar a cashless payment system for events, there were a few key criteria that drove my decisions for the overall architecture of the system.

Fundamentally, FastBar is a payment system designed specifically for event environments. Instead of using cash or drink tickets or clunky old credit card machines, events use FastBar instead.

There are 2 key characteristics of the environment in which FastBar operates, and the system that provide that drive almost all underlying aspects of the technical architecture: internet connectivity sucks, and we're dealing with people's money.

Internet connectivity at an event sucks

Prior to starting FastBar, I had a side business throwing events in Seattle. We'd throw summer parties, Halloween parties, New Year's Eve parties etc… In 10 years of throwing events, I cannot recall a single event where internet worked flawlessly. Most of the time it ranged from "entirely useless" to "ok some of the time".

At an event, there are typically 2 choices for getting internet connectivity:

  1. Rely on the venue's in-house WiFi

  2. Use the cellular network, for example a hotspot

Sometimes the venue's WiFi would work great in an initial walkthrough… and then 2000 people would arrive and connectivity goes to hell. Other times it would work great in certain areas of the venue, then we tested it where we wanted to setup registration, or place a bar, only to get an all too familiar response from the venue's IT folks: "oh, we didn’t think anyone would want internet there".

Relying on hotspots was just a bad: at many indoor locations, connectivity is poor. Even if you're outdoors with great connectivity, add a couple of thousand people to that space, each of them with smartphones hungry for bandwidth so they can post to Facebook/Instagram/Snapchat, or their phone just decides now is a great time to download that latest 3Gb iOS update in the background.

No matter what, internet connectivity at event environments is fundamentally poor and unreliable. This is something that isn't true in a standard retail environment like a coffee shop or hairdresser where you'd deploy a traditional point of sale, and it would have generally reliable internet connectivity.

We're dealing with people's money

For the event, the point of sale system is one of the most critical aspects - it effects attendee's ability to buy stuff, and the events ability to sell stuff. If the point of sale is down, attendees are pissed off and the event is losing money. Nobody wants that.

Food, beverage and merchandise sales are a huge source of revenue for events. For some events, it could be their only source of revenue.

In general, money is a very sensitive topic for people. Attendees have an expectation that they are charged accurately for the things they purchase, and events expect the sales numbers they see on their dashboard are correct, both of which are very reasonable expectations.

Reliability and Resiliency

Like any complicated, distributed software system, there are many non-functional requirements that are important to create something that works. A system needs to be:

  • Available

  • Secure

  • Maintainable

  • Performant

  • Scalable

  • And of course reliable

Ultimately, our customers (the events), and their customers (the attendees), want a system that is reliable and "just works". We achieve that kind of reliability by focusing on resiliency - we expect things will fail, and design a system that will handle those failures.

This means when thinking about our client side mobile apps, we expect the following:

  • Requests we make over the internet to our servers will fail, or will be slow. This could mean we have no internet connectivity at the time and can't event attempt to make a request to the server, or we have internet, but the request failed to get to the server, or the request made it to our server but the client didn't get the response

  • A device may run out of battery in the middle of an operation

  • A user may exit the app in the middle of an operation

  • A user may force close the app in the middle of an operation

  • The local SQLite database could get corrupt

  • Our server environment may be inaccessible

  • 3rd party services our apps communicate with might be inaccessible

On the server side, we run on Azure and also depend on a handful of 3rd party services. While generally reliable, we can expect:

  • Problems connecting to our Web App or API

  • Unexpected CPU spikes on our Azure Web Apps that impact client connectivity and dramatically increase response time for requests

  • Web Apps having problems connecting to our underlying SQL Azure database or storage accounts

  • Requests to our storage account resources being throttled

  • 3rd party services that normally respond in a couple of hundred milliseconds taking 120+ seconds to respond (that one caused a whole bunch of issues that still traumatize me to this day)

We've encountered every single one of these scenarios. Sometimes it seems like almost everything that can fail, has failed at some point, usually at the most inopportune time. That's not quite true, I can mentally create some nightmare scenarios that we could potentially encounter in the future, but these days we're in great shape to withstand multiple critical failures across different parts of our system and still retain the ability to take orders at the event, and have minimal impact to attendees and event staff.

We've done this by focusing on resiliency in all parts of the system - everything from the way we architect to the details of how we make network requests and interact with 3rd party services.

Processing an Order

To illustrate how we achieve resiliency, and therefore reliability, let's take a look at an example of processing and order. Conceptually, it looks like this:

FastBar - Order Processing - Conceptual.png

The order gets created on the POS and makes an API request to send it to the server. Seems pretty easy, right?

Not quite.

Below is a highly summarized version of what actually happens when an order is placed, and how it flows through the system:

There is a lot more to it that just a simple request-response. Instead, it's a complicated series of asynchronous operations and a whole bunch of queues in between which help us provide a system that is reliable and resilient.

On the POS App

  1. The underlying Order object and associated set of OrderItems are created and persisted to our local SQLite database
  2. We create a work item and place it on a queue. In our case, we implement our own queue as a table inside the same SQLite database. Both steps 1 and 2 happen transactionally, so either all inserts succeed, or none succeed. All of this happens within milliseconds, as it's all local on the device and doesn't rely on any network connectivity. The user experience is never impacted in case of network connectivity issues
  3. We call the synchronization engine and ask it to push our changes
    1. If we're online at the time, the synchronization engine will pick up items from the queue that are available for processing, which could be just the 1 order we just created, or there could be many orders that have been queued and are waiting to be sent to the server. For example if we were offline and have just come back online. Each item will be processed in the order that it was placed on the queue, and each item involves its own set of work. In this case, we're attempting to push this order to our server via our server-side API. If the request to the server succeeds, we'll delete the work item from the queue, and update the local Order and OrderItems with some data that the server returns to us in the response. This all happens transactionally.
    2. If there is a failure at any point, for example a network error, or a server error, we'll put that item back on the queue for future processing
    3. If we're not online, the synchronization engine can't do anything, so it returns immediately, and will re-try in the future. This happens either via a timer that is syncing periodically, or after another order is created and a push is requested
    4. Whenever we make any request to the server that could update or create any data, we send a IndempotentOperationKey which the server uses to determine if the request has been processed already or not

The Server API

  1. Our Web API receives the request and processed it
    1. We make sure the user has permissions to perform this operation, and verify that we have not already processed a request with the same IdempotentOperationKey the client has supplied
    2. The incoming request is validated, and if we can, we'll create an Order and set of OrderItems and insert them into the database. At this point, our goal is to do the minimal work possible and leave the bulk of the processing to later
    3. We'll queue a work item for processing in the background

Order Processor WebJob

  1. Our Order Processor is implemented as an Azure WebJob and runs in the background, constantly looking at the queue for new work
  2. The Order Processor is responsible for the core logic when it comes to processing an order, for example, connecting the order to an attendee and their tab, applying any discounts or promotions that may be applicable for that attendee and re-calculating the attendee's tab
  3. Next, we want to notify the attendee of their purchase, typically by sending them an SMS. We queue up a work item to be handled by the Outbound SMS Processor

Outbound SMS Processor WebJob

  1. The Outbound SMS processor handles the composition and sending of SMS messages to our 3rd party service for delivery, in our case, Twilio
  2. We're done!

That's a lot of complexity for what seems like a simple thing. So why would we add all of these different components and queues? Basically, it’s necessary to have a reliable and resilient system that can handle a whole lot of failure and still keep going:

  • If our client has any kind of network issues connecting to the server

  • If our client app is killed in any way, for example, if the device runs out of battery, or if the OS decided to kill our app since we were moved to the background, or if the user force quits our app

  • If our server environment is totally unavailable

  • If our server environment is available but slow to respond, for example, due to cloud weirdness (a whole other topic), or our own inefficient database queries or any number of other reasons

  • If our server environment has transitory errors caused by problems connecting with dependent services, for example, Azure SQL or Azure storage queues returning connectivity errors

  • If our server environment has consistent errors, for example, if we pushed a new build to the server that had a bug in it

  • If 3rd party services we depend on are unavailable for any reason

  • If 3rd party services we depend on are running slow for any reason

Asynchronicity to the Max

You'll notice the above flow is highly asynchronous. Wherever we can queue something up and process it later, we will. This means we're never worried if whatever system we're talking to is operating normally or not. If it's alive and kicking, great, we'll get that work processed quickly. If not, no worries, it will process in the background at some point in the future. Under normal circumstances, you could expect an order to be created on the client and a text message received by the customer within seconds. But, it could take a lot longer if any part of the system is running slowly or down, and that's ok, since it doesn't dramatically affect the user experience, and the reliability of the system is rock solid. .

It's also worth noting that all of these operations are both logically asynchronous, and asynchronous at the code level wherever possible.

Logically asynchronous meaning instead of the POS order creation UI directly calling the server, or, on the server side, the request thread directly calling a 3rd party service to send an SMS, these operations get stored in a queue for later processing in the background. Being logically asynchronous is what gives us reliability and resiliency.

Asynchronous at the code level is different. This means that wherever possible, when we are doing any kind of I/O we utilize C#’s async programming features. It's important to note that underlying code being asynchronous doesn’t actually have anything to do with resiliency. Rather, it helps our components achieve higher throughput since they're not tying up resources like threads, network sockets, database connections, file handles etc… waiting for responses. Asynchronity at the code level is all about throughput and scalability.


When you're building mobile applications connected to the cloud, reliability is key. The way to achieve reliability is by focusing on resiliency. Expect that everything can and probably will fail, and design you system to handle these failures. Make sure your system is highly logically asynchronous and queue up work to be handled by background components wherever possible.

Choosing a Tech Stack for Your Startup - Part 1: Client Apps

When I started FastBar in 2014, one of the first major decisions I needed to make was: which tech stack we were going to use? If you're starting a tech company, chances are you'll need to make this decision as well. Hopefully my decision process and lessons learnt can help you as well. 

This is the first in a 2 part series - in part 1, we'll deal with the client side of our app. I'll explore the history and decision making process behind the technologies I chose, talk about where we are at today, along with lessons learnt.

 By way of quick background, FastBar provides a cashless payment system for live events - think concerts, music festivals, beer and wine events etc… When an attendee arrives at an event, they get a wristband that's connected to their credit card. At the bar, they tap to pay in less than a second, and at the end of the event we automatically close them out with their card on file and send them an electronic receipt. I've previously posted some more details on the beginning of FastBar and why I started it.

 Fundamentally, FastBar is a payment system for events and is composed of 3 key components: a Registration App, Point of Sale (POS) App and the backend website and API. Since I first started FastBar, the product has evolved significantly, and the technology stack along with it.

High Level Components

 From a high-level perspective, here's what FastBar looks like today:

FastBar - High Level Components.png
  • Registration App - this is an app used onsite at events by event staff to swipe credit cards and issue RFID (or more technically, NFC) wristbands to attendees. It's also used for a variety of other maintenance tasks required onsite at events, like adding a new wristband to a attendee's account, disabling existing wristbands, loading cash onto wristbands etc…

  • POS App - used at events by event staff to sell stuff to attendees - whether that’s beverages, food or merchandise. Event staff enter what the attendee wants to purchase, then the attendee taps their wristband against an RFID/NFC reader connected to the POS to complete their transaction

  • The backend Web App and API have 4 key areas:

    • Attendee area: this allows attendees to check their tab during and after the event, request an email receipt, and register themselves online before they arrive at the event

    • Event organizer area: otherwise known as our Event Control Center, this is used by event organizers to setup and manage all aspects of their event, including configuring products, pricing, menus and view all kinds of reports

    • Admin area: used by FastBar staff to administer the system

    • API: used by our Registration and POS Apps, along with some 3rd party systems that integrate with FastBar

In this post, we’re going to focus on the client side technologies:

FastBar Components - Client Hilighted.png

FastBar's Early Development and Client-Side Tech Choices

The first app we built was the POS app. The prototype was created at Startup Weekend, and we went with iOS primarily because one of the guys on the team had a background in iOS development and was familiar with it. Skillset dictated our choice.

Android vs iOS

After Startup Weekend, when we got serious about moving FastBar forward and turning it into a real company, we needed to decide what platform we were going to use for the POS app. We went with Objective-C on iOS primarily because iOS was the most prevalent and consistent tablet platform on the market.

At the time (mid 2014), Android commanded a higher share of the global market for tablets and mobile devices 48% vs 33% for iOS:

However, when you narrow that to just tablets, iOS was significantly in front at 71%:

Narrow that down even further to North America, the primary market we cared about, and iOS is even more dominant with nearly 77%

Incidentally, here's how the tablet market share in North America has changed in the past 5 years:

In other words, not much.

Not only was iOS dominating in market share, but it was far less fragmented. As of June 2014, there were only 7 different iOS devices, all manufactured by the same company, Apple:

  • iPad

  • iPad 2

  • iPad 3rd gen

  • iPad Mini

  • iPad 4th gen

  • iPad Air

  • iPad Mini 2

The only real difference between these devices from a development perspective was the screen size and resolution. Building apps to run on all of these devices was straightforward.

By contrast, the Android market was highly fragmented with hundreds or probably thousands of devices available from tons of different manufacturers. Trying to build an application that will work consistently across all Android devices was a lot harder.

Given iOS was dominant and consistent, this meant:

  1. Plenty of dev - There were plenty of developers around who knew how to build apps for iOS

  2. Customer familiarity - our potential customers often times already had iOS devices. It would be easier for us to convince them to use a platform they were already familiar with rather than introduce something new into their environment

  3. Large rental market - there was a large market for renting iPads, meaning we didn’t need to incur a lot of up-front capital expense, instead, we could own a handful of devices and rent the additional ones we needed for an event

  4. Lots of accessories - there was a lot of choice in accessories for iPads, like point of sale stands. Less so for Android devices

From a pricing standpoint, although it was possible to find really cheap Android tablets (around $50-80), by the time you went for something that was higher quality and similarly speced to the iPad, you were looking at a similar price point.

With the hardware decision made, we continued to evolve the first prototype and stuck with Objective-C as the language of choice. Later in 2014 Apple release Swift, so we started to build some parts of the app in Swift as well.

The First Live Event

In August 2014 we executed our first live event where we processed payments. This made use of our POS App, and to register attendees we used a web-based registration app running on a laptop with a USB connected RFID reader. We quickly realized that the web-based app running on a PC was not a good option for us. We needed to be mobile, and a phone form factor was going to be a much better solution. 

At that point we set out to build the first version of our Registration app, again using Objective-C and Swift running on iOS.

Native or Not?

In early 2015 we decided we needed 3rd app, an Attendee app, that would be used by attendees to check their tabs during the event. Since this app was attendee facing it needed to be deployed on their own devices and we had no control over what hardware they'd be using (unlike the POS and Registration apps), that meant it needed to be multi-platform. Building for Android and iOS would allow us to cover well over 90% of the market, which was good enough for us as a startup. Back at the time Windows Phone still kind of existed, but was on the decline, and quite frankly just not relevant for us.

We had a decision to make: do we go native and build separate apps for each platform, or use a use a cross-platform technology that would allow us to target both Android and iOS. Both options had their pros and cons. If we went native, we'd have full access to the underlying platform, and be able to create the best experience possible for users. However, we'd be maintaining 2 different code bases, which is tough, especially for a resourced-strapped startup. If we went cross-platform, we'd have a single code base to maintain so development would be quicker. On the flip side, cross-platform technology wasn't as evolved back then as it is now, couple that with the face that no matter what, you will never be able to create the most optimized experience using a cross-platform technology compared to native tech. By definition, you're catering for the lowest common denominator.

Betting on Xamarin

We decided that the benefits of some kind of cross-platform tech far outweighed the cons, and our requirements were fairly basic and something that could fit within the realm of what cross-platform tech was good for, ie a fairly standard "business" app which primarily required forms and data synchronization. In our case, we also needed to interact with local hardware.

We decided to give Xamarin a shot. If you're not familiar with it, Xamarin allows you to build native applications for iOS (using Xamarin.iOS) or Android (using Xamarin.Android) using C# and .NET. The cool thing about Xamarin is that if you factor your application correctly, you can re-use any code that is non-platform specific. For example, a lot of our app dealt with network communication, data synchronization to the server and data manipulation locally (ie reading from and writing to our local SQLite database). Essentially all of that could be re-used across both iOS and Android without changes. In addition, in mid-2014 Xamarin released a technology called Xamarin Forms which allows you to build UI screens in a cross platform manner. This meant we could write a screen once, and it would run on iOS and Android without us having to build separate screens for both platforms. Xamarin Forms is also smart enough to render controls in a platform specific manner, for example, when we define a date picker on the screen, it will look like a native iOS date picker when running on iOS, and a native Android date picker when running on Android.

Essentially, Xamarin allowed us to build an app that was both native and cross platform - it's kind of like the best of both worlds.

I was initially cautious of Xamarin - I'd had some previous negative experiences when using an earlier version of Xamarin a couple of years prior, but one of the guys on our team had used it recently, was familiar with it, and assured us it had evolved significantly in the last couple of years.

In addition, it would allow us to use the same language on the client apps that we were using on the backend, C# (more on that in part 2 of this blog post), and that meant it was easier for developers who'd be working on the client and server to transition between codebases given the language was the same.

Our bet on Xamarin proved to be a good one. We rolled out the Attendee app on both Android and iOS simultaneously. It took us around 20% extra time to release apps for both iOS and Android vs what it would have taken for us to build for a single platform if we were doing it natively.

First Major Rebuild for Registration and POS

Towards the end of 2015, we decided it was time for our first major re-build. We were constantly evolving the code bases for both the POS and Registration Apps, but were starting to find it was harder for us to add new features and harder to find and fix bugs. Both of these apps had evolved from prototype code, and were never designed to last as long as they had. We were in a position where were just couldn't move as fast as we wanted to.

Our experiment with Xamarin for the Attendee app had been a success, so we felt confident picking Xamarin as our primary development platform moving forward. The Registration app was the highest priority for us to rebuild, as it needed some significant changes. We decided to tackle it first, with a view to building as many re-usable components as possible for use in the POS, when we got around to rebuilding that as well.

We rolled out the first version of our new Registration app in early 2016 and continued to evolve it throughout the year, eventually sunsetting the legacy Registration app when the new one had proven itself at live events and had subsumed all features of the legacy version.

Towards the end of 2016, we started work on the fully re-built version of the POS app, which initially rolled out early 2017. Similarly, we maintained the legacy version of the POS until the re-built version was battle-tested and ready for prime time.

Side note: Sunsetting the Attendee App

Also towards the end of 2016 we decided to sunset the Attendee app. We always knew that we could never force attendees to download an app at an event - it's a poor user experience to require this, and also impractical in an event environment where connectivity could be scarce or nonexistent, ie 5000 people each trying to download and 80Mb app with little to no connectivity is a nonstarter.

The key features that an attendee needed were straightforward, so we decided to focus on just delivering those features via a mobile friendly website and SMS, which was a simple and more universally accessible solution.

Code Reuse Across Multiple Apps

We found that reusing code across both the Registration and POS app was tougher than we originally thought. It wasn't until the Registration app was re-built and we started work on the new POS app that we realized a lot of decisions we made in the implementation of the Registration app were not conducive to reuse in another app, so some refactoring was required.

Code reuse is an interesting beast - on one hand we wanted to move quickly on the development of the Registration app and didn’t want to invest too much time in over-complicating the design or over genericizing things that might need to be reused one day, or might not. That violates the YAGNI principle: Ya Aint Gunna Need It. On the other hand it doesn't make sense to duplicate logic and components in two different apps, which just leads to inconsistency from the user experience and is harder to maintain and debug . That violates the DRY principle: Don't Repeat Yourself.

It's a tough balance to find the right level. Over time, we got better at determining what should be reused across an app and how to structure that code, and what we should keep separate for each app. It's something that we're constantly monitoring as we build new features.

Side note: Microsoft's Acquisition of Xamarin

In February 2016 Microsoft acquired Xamarin - I felt that was a good sign in terms of the longevity of the technology. Since then, I think Xamarin has continued to develop nicely, and looking back, I'm happy with the bet we made on Xamarin.

Where We're at Today

Today, FastBar has 2 client apps: the Registration app:

And the POS app:

They're both built in C# using Xamarin, and both target iOS only (at the moment).

Almost all screens are built using Xamarin Forms, except for 1 screen in the POS, the main screen:


(Note: it is possible to have screens built in Xamarin Forms and screen built in Xamarin iOS or Android exist side by side in the one app)

Originally, this screen was built in Xamarin Forms, but we found the responsiveness when tapping the screen wasn't quite where we wanted it to be. Sometimes, when tapping the screen in quick succession taps would be missed and those items would not end up in the shopping cart. We decided to build natively in Xamarin iOS to achieve the best possible performance. It seems slightly quicker and more responsive than it's Xamarin Forms counterpart. Having said that, I'm sure Xamarin Forms has improved, and nowadays it might be as performant as the native implementation. What we have now works well, so there is no need for us to change it.

Since both apps are quite similar, both visually and from an underlying logic standpoint and how they interact with the server, we have a high-degree of code and UI screen re-use across both of the apps.

As of the date of this post, we don’t yet target Android, although it is reasonably trivial for us to do so.

We don’t have an Attendee app any more, it’s not really necessary, although may choose to re-introduce one in the future. We may also add an Event Organizer app that will allow Event Organizers to manage their events when the event is in operation. Today they can fully manage their event from our mobile friendly website, but a app running locally would allow us to provide a better experience, especially operating at event environments with poor internet connectivity.

Lessons Learnt

Optimize for Productivity

When choosing what technology you want to use for your startup, optimize for productivity.

Let's say there are 2 technology stacks: A and B. Your team has background and experience in A and thinks it will take 12 weeks to build your minimal viable product (MVP). Your well-meaning, but technologically religious friend, let's call him Bob, says "You're crazy, why would you choose A, B is so much better. I could build it in 8 weeks using B!".

Maybe Bob is right.

Maybe B is way more productive than A.

Maybe, given 2 capable teams, building this solution out using A will take 12 weeks and B will take 8 weeks.

But probably it won't.

And it will almost certainly will take your team, familiar with A, longer than 8 weeks to build something in B, because they need to get up to speed with the tech.

When thinking about productivity, definitely factor in things like language productivity, frameworks available etc… and also be sure to factor in the experience and skill set for your team. Choose what's going to be most productive for you.

Choose Something Popular

Quite frankly, it doesn't really matter what technology stack you choose, within reason.

For the most part, a good team is going to be able to build out your solution in Objective-C or Swift (native iOS) or Java (native Android) or Xamarin or React Native or Ionic or Phone Gap (cross-platform-ish technologies). Just make sure you choose something that is popular, meaning:

  • It is used by many applications

  • You can easily find and hire developers who understand it

  • It is actively maintained

Popular frameworks have companies, or more commonly, an open source community, behind them who keep building new features, keep fixing bugs, keep plugging security holes etc… all on your behalf! You don’t want to take this burden on by yourself, you have better things to do. Choose a technology that's popular.

Choose the Simpliest Thing That Works

Follow the old acronym, KISS: Keep It Simple Stupid.

Engineers like to engineer. It's in their nature. This means it very easy for them to over-engineer a solution. Choose the simplest solution that will solve your problem, and work with it for as long as possible.

For us, we ended up sunsetting one of our apps, the Attendee app, when we realized it was not the simplest solution possible. It was extra code to maintain that we didn’t really need, it was not something we could force on users, therefore we needed a more universally accessible solution, like a mobile friendly website. Given we needed the website anyway, did we really need the app? It was unnecessary, so we killed it.

Now, we may bring it back at some point in the future if it becomes necessary or advantageous, but for the moment, the mobile web solution we have is simple, it's easy, and it works. Choose the simplest thing that works.

Favor Cross-Platform Tech

If you're building for multiple platforms, do yourself a favor and use some cross-platform tech.

If you know for certain that only care about Android and your team has experience in Java-based Android development, awesome, refer to "Optimize for Productivity" above and go for it.

Likewise, if you're only building for iOS and you've got the experience in Swift or Objective-C, good for you, fire up Xcode and get crackin'.

But if you're looking to build apps for multiple platforms, like iOS and Android, do yourself a favor and go with some kind of cross platform tech so you can get as much code reuse as possible. We went with Xamarin and it has worked well for us, but whatever you choose, you don’t want to be implementing the same features 2 different ways in 2 different platforms, dealing with 2 different sets of bugs to fix  etc… That's a nightmare, and you've got better things to do.

Also, if you think there is a high probability that you'll need to go for multiple platforms in the future, do yourself a favor and choose some cross platform tech to begin with. It will probably be just as quick to build your app as it would be to do it natively, and now you're got the added flexibility that you can add that other platform if and when you need to later on. Favor cross-platform tech.

Stay tuned for part 2 of this series where we'll explore server side technology choices…