On Marketing Systems

A couple of years ago I was working in an e-commerce specialized development agency. While I was there my boss assigned me a new client that needed a systems integrator. The client wanted to connect their webshop with the new cloud marketing systems for which they had payed a hefty licencing fee.

Integrating with a complex system

I took lead on this mini-project. It encompassed all of the layers of an e-commerce clustered monolith. I added a new JavaScript library on the frontend to capture customer actions. Of course, I needed to customize the library so that it could work properly with the client’s setup. The frontend code dispatched all captured events to the backend. The same code would tag each user with an id. Based on this id , we could attribute each event to a particlar user or session.

Typically Tracked User Actions

Searches
Page Views
Category Views
All cart operations – addition, removal, bulk and single
All order operations – payment, cancellation, refund, return, delivery

Next, I amended the out-of-the-box backend Java code which received and processed events from the frontend. Thankfully. functional programming had arrived to Java since version 8. Indeed, such a programming style proved very useful in writing pipeline code for processing events. Besides improved readability, I was able to formulate a solution more easily. Finally, the code validated the events and enriched them with data. Then it would batch them and ship them off to the marketing system.

Enter machine learning

The frontend event stream fueled machine learning models on the marketing systems side. The information includes things like items searched, browsed and purchased. Users automatically cluster along multiple dimensions. The cluster assignments feed back into the e-commerce system. We tailored the content according to each users cluster assignment. Each group (cluster) of users could see different content. Finally, email campaigns are also targeted per group. Thus, they increase the likelihood of the users getting relevant content-

The marketing systems receive other data streams at the same time. Daily data dumps over FTP contained product and pricing data. Combined with user clustering, the storefront displayed product recommendation components. These components would yield recommendations based on the ranking algorithm that was configured for that component. Be sure to check the list below for a summary of all the system inputs and outputs. In addition, there is a second list below that one that details the user actions that are tracked on the storefront.

Marketing systems inputs and outputs

Input streams

User Actions over REST
Product Data over FTP
Price Data over FTP
Category Data over FTP

Resulting features

User clustering
Product Recommendations
E-mail customization
CMS customization

Takeaways

The whole experience left a double impact on me. On the one hand it is easy to see how such systems can benefit e-commerce. Driving sales becomes easier and more rational. The user receives less spam and more useful information. It’s a game changer and adoption is guaranteed to be widespread. On the other hand, it raises significant questions regarding users’ data and privacy. This is no longer an issue just of the EU and GDPR. Other big players – countries, trading blocks – are also looking into privacy and data management legislation.

The political forces behind such legislation are considerable. No wonder, it’s easy to understand why customers could get uneasy regarding these features. With proper discussion and as practice takes shape, the downsides will be reigned in. However, it can still be a cause of headaches for developers and operations. They need to be able to produce and delete all of the users data at request.

Ivan Šarić

Engineering Blog