Logistics is one of the most advanced areas in ICT and related technologies today. All recent issues arising – microservices, coexistence with legacy systems, process streamlining, big data cleansing, data streaming – are currently dealt with using code-managed infrastructures.
New services, such as AI-based software, can easily be introduced in this open architecture, its further development securely in the hands of experienced DevOps-focused engineers who constantly target their activity on increasing company value through continuous, yet secure, delivery of new releases for existing software and brand-new code.
That makes logistics and large retail/supermarket chains probably the best places for a developer to gain experience today.
Retail and logistics, where microservices rule
This is also the train of thought at ShopFully, where data sits at the heart of business. ShopFully’s vision is to become the world-leading tech company in drive-to-store and proximity marketing solutions.
What is this company’s business and why is it relevant to a developer’s career? ShopFully is a tech company, and a leader in drive-to-store solutions. The company’s mission is to make local shopping easier. The company helps more than 40 million consumers globally to save time and money through in-depth information on nearby stores.
It is a reliable tech partner to over 700 retailers and brands at an international level. These are inspiring numbers for developers who want to grow in a real-world environment.
ShopFully is also a place for retailers and brands to engage with shoppers when they are most open to explore new shopping opportunities. According to their website presentation, ShopFully’s people are the omnichannel shopping experts, measuring and predicting billions of paths from ad exposure to physical store visits.
The company offers a full cycle, including software and marketing development. For hundreds of retailers, this platform offers the best way to reach millions of shoppers with promotions and generate measurable store visits.
ShopFully offers mobile-to-store advertising and profiling solutions specially designed for brands to influence customer behaviour.
The company normally hires qualified developers, currently prioritising full stack competencies (React, Node.js) and classic backend competencies (PHP, Amazon AWS) with a mid- to senior level of expertise and a minimum of 3 years of experience in that role.
PHP runtimes for the microservice environment
ShopFully offers high-level retail software solutions for the biggest global chains. Their ICT offering hosts both legacy and modern solutions in a well-orchestrated workflow management system.
Today’s companies need to update their existing portfolio, adopting a modern and consistent app model for all phases in the app life-cycle. This demands a continuous evaluation of the overall portfolio, in order to schedule updates for good, but old, software.
That leads to a coexistence of both worlds – old and new – in this transformational software paradigm. A migration of classic ICT solutions to modern architectures is underway in ShopFully, together with state-of-the-art, serverless, microservices-based applications.
Most of the old software was written with PHP language. PHP applications can be included in the world of microservices thanks to mechanisms provided by third-party applications.
The existing PHP-based backend relies on Amazon AWS and Bref to run and deploy serverless PHP applications.
Bref generates open source, production-ready PHP runtimes for AWS Lambda functions, defining itself as “a higher level stack for a better view of the cloud”.
Creating and running modern and scalable PHP applications is not always as simple as it should be. By using serverless technologies like AWS Lambda, the focus turns to development, with servers forgotten.
Bref is an open source project that brings full support for PHP and its frameworks to AWS Lambda. It can be overwhelming getting started with the serverless paradigm, if your starting-point is a classic backend approach.
Bref integrates with the open source Serverless Framework for configuration and deployment. Powerful, yet simple, it offers the best of both worlds.
As well as the Serverless Framework’s documentation and ecosystem, Bref provides complete documentation and examples for PHP applications.
This presents a viable method of integrating legacy applications in a more modern execution environment. Serving new needs, or old needs in a new way, is a demanding challenge that requires a novel approach to data management through many services.
New elements are developed with a different full-stack programming flux that is based on a classic Javascript-powered Node.js/React approach. Some stages of this process are commonly referred to as a saga.
What is Saga?
Software solutions are very innovative in this large-scale, evolving environment. The overall workflow is the basis on which every piece of software is integrated into the existing portfolio, or modified according to new needs.
The workflow used allows sellers to place purchase orders on behalf of customers and automatically associates and creates the bid campaigns that are the core of the business model.
Collected data have to be leveraged in the way most useful to subsequent needs, losing no part of their identity.
The workflow interacts with different information systems. Communications with separate systems must then be guaranteed while the integrity of the data is maintained. Classic transactional databases are not the reference choice for this task family, that title belongs to distributed systems.
In the microservice-based approach, one structural issue is keeping data consistent across multiple services. Transactions often have a long journey to completion, crossing multiple databases and communicating with multiple services that may be temporarily unavailable.
Local ACID transactions are not always of help in this context. In fact, according to the AWS page on step functions, “because these microservices cannot use a single atomicity, consistency, isolation, durability (ACID) transaction, you might end up with partial transactions. In this case, some control logic is needed to undo the transactions that have already been processed”.
The classic approach is the so-called 2PC (two-phase commit). This theoretically solves the problem if everything goes to plan, but this particular case is not the standard scenario in the microservice-based world.
How to manage a sequence of local transactions is the issue at hand. The distributed saga pattern is typically used for this purpose. The sequence of events involved is normally referred to as a “saga”.
It’s amusing to note that the original meaning of the word refers to long, epic stories written mainly in Iceland between 1120 and 1400. In later times, the term is extended to cover the story of a family and its cross-generational problems, as detailed in a report – and this leads us to an explanation of the modern, technological use!
Returning to transaction-solving, the inclusion of a saga gives rise to the need for management. There are two common ways to achieve saga management, choreography and orchestration.
Each local transaction publishes domain events that trigger local transactions in other services, dealing with a choreography.
An external object known as ‘the orchestrator’ rules all the local transactions to be executed by every called service.
This kind of process management leads to great results, as well as ensuring rollback if required. This makes saga-based development look like one of the best ways to ensure the consistency of data in a distributed architecture.
ShopFully’s management chose to tackle this issue by using the Saga Pattern. Using AWS as a Cloud provider, the natural choice was to use AWS Step Functions to implement the Saga Pattern.
Relating unrelated processes
Originally, the purchase flows of ShopFully’sproducts and their delivery were disconnected from each other, with information being passed manually, without automation.
When the two flows were integrated, they involved heterogeneous systems and therefore the problem of maintaining the involved data’s integrity arose. One way to handle this is to use the saga pattern.
When you use the saga pattern, “every service that performs a transaction publishes an event that triggers subsequent services to perform the next transaction in the chain. This continues until the last transaction in the chain is complete. If a business transaction fails, saga orchestrates a series of compensating transactions that undo the changes that were made by the preceding transactions”.
In brief, then, for each operation performed on a system, there is a compensating transaction that cancels the operation performed if something fails on any of the systems involved.
A good solution to the problem involves:
- Managing long-running transactions
- Data consistency maintained by the saga pattern
- Serverless and fully managed approach
- Infrastructure as code
By studying the state of the art in terms of fully-managed solutions, we analyzed the AWS Step Functions:
- Business logic workflow orchestration service;
- Orchestration of distributed services supporting native integrations (less code required);
- Run a Lambda, queue a message on Amazon Simple Queue Service (SQS), wait for the result and pass it to another service.
ShopFully – having used AWS for years – made it natural that they should prefer a solution from this provider. The company’s previous experience with Serverless Framework made it the preferred choice of tool for deploying and managing serverless architectures.
Backward recovery
Backward recovery is an advanced solution. The data journey of a ShopFull customer comprises many different solutions that must be kept separate. The information flux crosses many hot points, each of which may have a valid or invalid transaction. Each step must be retraced if one step is invalid.
ShopFully decided to leverage the AWS service infrastructure. Various articles reported examples of similar use cases. An excellent piece of work from Yan Cui allowed the implementation of Step Functions on a serverless framework thanks to the Serverless Step Functions plugin.
ShopFully found that it was possible to manage Step Functions via the Serverless Framework plugin (maintained by Yan Cui himself, among others), and therefore implemented an architecture based on Step Functions.
Orders are made by ShopFully’s Order & Campaign Manager team on behalf of customers. Data consistency is ensured thanks to the efforts of ShopFully’s staff, using Saga Patterns.
A transaction process’ success story
- CreateOrUpdate: the execution plan is quite different both from a workflow point of view and from an internal operations point of view, so a clear separation between create and update cases was made.
- PrepareCreate: Step Functions work by ‘moving’ an execution payload in the flow, in this step it is enriched with necessary information during execution
- CreateFlyerGroup: in this step, real logic takes centrestage: flyergroup is a domain object used internally in ShopFully for campaigns setup; it is created in this step.
- FootfallCreateOrNot: decisional step; the decision whether or not the subsequent step (CreateFootfallCampaign) must be taken for the specific execution is made now.
- CreateFootfallCampaign: in this step the Campaign for Footfall activation is configured; this primarily consists of the creation of another domain object in another systems
- CreateCampaign: in this step the real Campaign object is created; this serves as an index for all domain objects created, and encapsulates various information from all the systems involved
- UpdateFlyerGroupMetadataOnCreate: a data consolidation step for working internals
- SuccessCreateForm: the final step, in which the operation as a whole is marked as a success
- End
- CreateOrUpdate: as previously
- PrepareUpdate: the same as PrepareCreate (above), only this time in the update branch with some different inner workings
- UpdateFlyerGroup: as for CreateFlyerGroup, except as with PrepareUpdate, we are in update mode, with various internal differences
- UpdateFootfallCampaign: replicates CreateFootfallCampaign, although again, in update mode with various internal differences
- RollbackFootfallCampaign: in this execution an error occurred during UpdateFootfallCampaign execution, starting the cascading rollbacks, with RollbackFootfallCampaign being the first: in this step the FootfallCampaign object will revert to its original status, which was snapshotted beforehand
- RollbackFlyerGroup: in this step the FlyerGroup object will revert to its original status, which was snapshotted beforehand
- FailUpdateForms: the final step, where the operation as a whole is marked as a a failure
- End
Conclusions
A brave new world of ICT solutions is opening up, on the back of cloud-based patterns. Backward recovery on Amazon services is one of the examples that best shows how the real world is being modeled by software in today’s world.
The need for solutions that can be correctly implemented in an ICT environment, hosting legacy and modern applications at the same time, is the challenge that those working in software development chains must face up to now – and for a long time to come.
New patterns will require consideration in the next few years, and real-world environments such as logistics and retail-based activities ensure that old and new skills are performed in an updated context, giving new scope to the developers job.
