Ehsan Zanjani
Ehsan Zanjani Senior Application Architect
14. February 2019 in

Technology English Dev

How About Some Real-Time Data Transfers From Back-end to Front-end?

Separation has an impact on every aspect of software design. Whenever there is some element of separation, there should be a way to sync data between those separated parts. When it comes to the Spryker Commerce OS architecture, these separated parts are called Yves (front-end) and Zed (back-end); which is why we need a data transferring process between those two. I’m going to explain our very new approach of data synching mechanics, which is called Publish and Synchronize. 


Zed
as the back-end holds the core logic and interaction with the database while providing APIs for other parts of the system, like Yves.

Yves as a front-end is the customer-facing interface and delegates user requests to Zed. It shows elements like product details, content and navigation. Yves conceptually does not have any connection to the Zed database, but it can connect to other storages including NoSQL databases like Redis, Elasticsearch or 3rd-party endpoints. The reasons for establishing this separation are rooted in ensuring better performance, scalability, and security.

 

Start by Clarifying the Problems

Data-syncing processes are not as easy and straightforward as they may sound. Sometimes data gets out of sync unexpectedly during the data transfer between the two layers. As a result, data may get lost or crash with an untraceable path.

  • Data consistency is one of the main points to analyze when you have two sources of truth. Inconsistencies might occur when one source of data got updated and another did not. This could have several reasons including missing data, errors, or communication issues.

  • Latency is another important subject which can affect shop performance and revenue in the long run. The syncing time should be fast enough to not affect business logic and selling strategies.

Spryker’s Syncing Method and Solution

Analyzing all the requirements for a fresh approach, obviously, the most essential part of the process was a Queue system to send and receive the messages between the components. The next requirement was Event/Listener for watching the system changes. Lastly, we needed to design Entities smart enough to run these actions. The rest of the operation was about connecting these components together and making Publish and Synchronize (P&S) work.

In the following picture, you can see the overall architecture of Publish and Synchronize. We will go into further detail about this later.

Data_Transfer-02
The next challenge was to get all existing modules to work with the current mechanism as well as with the new approach, while not modifying all modules, nor injecting new codes into them. To achieve this goal, we came up with an abstract method ensuring all modules can handle the new approach based on minor changes. It is called the Event module.

The Event module allows developers to create custom events and register so many different listeners.These events enter the queue as simple messages where listeners are able to consume these Events. The great thing about this is Event modules can effectively listen to any changes across the system.

From here, every change was being tracked by these events. The next step was to create the right listeners for them. These listeners are responsible for aggregating the front-end data. When an event message has been triggered and is then consumed by a listener, it will run a query to take all related data for that specific entity model. Consequently, it saves the result in a json format to a special database table which is called Storage or Search. These tables contain exactly the same data, which the external resources expect it to have. Because of this, sometimes they are called Mirror tables. As soon as we store the results of the query in these tables, the first part of our mission, which we call Publish, is done.

The next part of this process was moving the aggregated results from the Mirror tables (storage and search tables) to the final endpoints. The results of previous parts were being saved into the database. Simultaneously, a copy of that data was being sent to the corresponding queues. Later, these messages were consumed by storage and search plugins and the data was transferred to endpoints. This process is called Synchronize.

 

What are the Benefits of Publish and Synchronize?

Just thinking about an accident in production makes all development and business teams scared especially if you end up losing highly valuable data which impacts important parts of the shop and consequently revenue. Using Publish and Synchronize takes only a couple of minutes to re-export all the data again, as the aggregated data was already stored in the Zed database.

Consistency with P&S is very strong. It is because of the data which is stored in special database tables that everyone working with the Spryker Commerce OS can now easily debug missing elements that have been lost along with the data. Speed is another important element in software architecture. Using the new P&S- approach, we reduced the update time to 1-2 seconds, which is not only fast but close to real-time.

As long as P&S is based on the queue system, it allows you to scale the processors in case you experience high amounts of data. The queue processors can be configured to run multiple instances of PHP processors per server.

Spryker is using transfer objects across the whole system. This is also supported by P&S. When developers look up data from Redis, they will know what elements are Transfer Objects.

 

Why do our Developers like Publish & Synchronize so much?

For one, it is very easy to learn and has a very clear process flow. Entities trigger events, and listeners will handle the rest. Developers do not need to write extra code for publishing data from back-end to front-end, as this is now handled automatically. Another perk is it is much easier to debug the publishing process. There are more isolated parts, which can be debugged independently from each other.

Ready for smooth data streams and updates? Get Publishing & Synchronizing!

 

Get all new updates straight to your inbox