Data and Event Tracking Component

We started the project by creating a story map to help us:

• Define the key components and features of the tracker.
• Identify and solve the main technical challenges, such as performance, speed, and scalability.
• Decide what events the tracker should collect and how it will do it.

Based on the results from the story mapping sessions, we created a list of functional requirements for the tracker. The functional requirements relate to the features and processes of the tracker.

We identified that the tracker would need to:

• Receive and handle multiple types of requests (impressions, clicks, conversions, etc.).
• Process requests and generate events with proper dimensions and metrics.
• Allow request types and preprocessing of events to be configured.
• Extend its basic functionalities with plugins by exposing its API.
• Expose generated events to specified collectors (plugins) and allow them to be attached to specific event types by configuration.
• Include built-in log and queue collectors.
• Include a built-in plugin for our budget management component, banker.
• Support request redirects.

The non-functional requirements of the project aren’t connected with building a working product, but are connected to the performance, scalability, security, delivery, and interoperability of the tracker.

We identified the following non-functional requirements of tracker:

• High availability (99.999%).
• High requests per second (2000 requests/s).
• Low requests processing latency (15 ms on average).
• Security
  • Privacy — e.g. ensuring we don’t expose personal data to third parties when we need to share it with other components.
  • Availability — ensure the platform won’t be impacted by a distributed denial-of-service (DDoS) from bad requests.
• Deployable to AWS, GCP, and Azure.
• Ability to integrate with custom plugins.
• Ability to integrate with other DSP components such as the banker.
• Platform scalability — the tracker needs to be able to handle large increases in events.
• Idempotency — processing the same logs many times in case of errors or temporary unavailability

We selected the architecture and tech stack for the tracker project by:

• Researching benchmarking tools used for testing the performance of the programming languages.
• Comparing different variations of the technology stack.

The main metrics we wanted to test were:

• Latency of requests
• Requests per second
• Error ratio
• Percentiles latency

Technology Stack Comparison

We decided to test the following programming languages and technologies:

• Golang (aka Go) because it’s growing in popularity and we were familiar with it.
• Rust because other development teams have used it to build trackers in the past.
• Python because we are very familiar with it.
• OpenResty = Nginx + Lua because it allows us to create a tracker using just an nginx HTTP server.

We decided to run initial benchmark tests on all the technologies using the benchmarking tools listed above, but later focused on running more tests using wrk2, gatling and locust.

All three tools were configured to allow for maximum elasticity.

The results and the chosen programming language

The two best performing programming languages were Nginx + Lua and Golang, and while they had similar results across all benchmarking tools, we chose Golang due to current market needs and its popularity.

The MVP Scoping Phase took 1 sprint (2 weeks) to complete.

With the MVP Scoping Phase completed and our architecture and tech stack selected, we began building the MVP of the tracker.

We built the tracker in 5 sprints (10 weeks).

Below is an overview of what we produced in each sprint.

Sprint 1

What we achieved and built in this sprint:

• Implemented basic functionality.
• Configured event types.
• Created benchmarks as a continuous integration (CI) step.
• Produced visual benchmark results.

Sprint 2

What we achieved and built in this sprint:

• Created and tested request redirects.
• Performed tests on the tracker.

Sprint 3

What we achieved and built in this sprint:

• Built a plugin for the log storage component.
• Enhanced configuration that allows us to configure any tracking paths..
• Produced the tracker’s documentation.

Sprint 4

What we achieved and built in this sprint:

• Built a plugin for our banker (a budget management component).
• Configured event extraction.
• Ran end to end tests.

Sprint 5

What we achieved and built in this sprint:

• Produced auto-generated documentation.
• Created a quickstart guide.
• Built docker images and pushed them into the internal registry.

Our Tracker component is designed to:

• Speed up the development phase of your custom-built AdTech or MarTech platform.
• Implement it into your existing AdTech or MarTech platforms to collect data.