Mobile Dashboard in near real time

I chose Klipfolio as a perfect visualization tool for operation reporting. 

There is overview of working architecture:

Using Amazon Redshift Instance I collect  data from adjust. Adjust sends data to ourt local database via callbacks. Than using pentaho data integration as ETL tool I take data and load it to the redshift. Job in pentaho runs every 10 minutes and incrementally update last 7 days. Considering mobile applications we should considering long period, because some phones can send data to mobile tracker with long delayed, that's why I prefer to take 7 days and update them with each load.

Data Flow

Using Pentaho data integration I create 2 transformations and put them in one job.

Run Pentaho DI

Pentaho home path: /home/dasboard/Downloads/data-integration

in order to start pentaha use terminal and run spoon.sh

Transformation 1: stg_adjust

This transformation extract data from mob.adjust for last 7 days and send it to s3://s3/webanalys/adjust_data in gzip format.

Transformation 2: sql_adjust

This transformation run sql queries:

--copy data from file to stg.stg_temp_adjsut

copy stg.stg_temp_adjust from 's3://webanalys/adjust_data'
credentials
'aws_access_key_id=AKIAIFNNQI73NUWZ7BZQ;aws_secret_access_key=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'
delimiter '|'
timeformat 'YYYY-MM-DD HH24:MI:SS'
gzip;

--delete data from stg.stg_adjust in order to write new fresh data
DELETE FROM stg.stg_adjust where stamp>= (SELECT MIN(stamp) FROM stg.stg_temp_adjust);
INSERT INTO stg.stg_adjust (SELECT * FROM stg.stg_temp_adjust);

--optimaze stg_adjsut table
analyze stg.stg_adjust;
vacuum stg.stg_adjust;

--delete from temp table all data
DELETE from stg.stg_temp_adjust;
vacuum stg.stg_temp_adjust;

Job: jb_adjsuttoAWS

Kitchen job

Pentaha allow as to schedule jobs via kitchen. Using crontab I run my job. Moreover, kitchen can perform only shell script. As a result I put my job in shell script:

#!/bin/bash
/home/dasboard/Downloads/data-integration/kitchen.sh -file=/home/dasboard/Downloads/pentaho_di_files/jb_adjusttoAWS.kjb -level=Minimal

When we successful load data to redshift, we can build new klip using database connection in klipfolio.

Just choose postgres database and fill credentials. Unfortunately, klipfolio has one disadvantage, you should create separate query for almost every klip. 

My job runs every 10 minutes. As a result we get fresh data and can quickly check health of mobile applications and their performance.

Klips: Installs, Average Session duration in sec, registrations, daily active unique users, weekly active unique users, monthly unique users and installs by channels:

Adjust Implementation

Going deeper with mobile tracking, I start to look at new mobile trackers. And chose one of the leader - adjust. In order to implement adjust to our mobile apps, we should add SDK to the mobile app.

First of all we should read docs:

• Getting Started https://docs.adjust.com/en/getting-started/
• Event Tracking https://docs.adjust.com/en/event-tracking/
• Tracke Generation https://docs.adjust.com/en/tracker-generation/
• Integration with facebook https://docs.adjust.com/en/facebook/
• Google ad tracking https://docs.adjust.com/en/google/

For mobile developers we should send link on github:

• Github repo https://github.com/adjust/sdks

Then add our application to adjust by typing App ID:

In addition we should describe screens of app and events. If we want to send additional dimensions or attributes we should use callbacks. Because due to personal data law, adjust don't store customer data.

• Working with callbacks https://docs.adjust.com/en/callbacks/

In order to create callback receiver, we should tell to system administrated, what fields will come from adjust. We can learn about default placeholders - https://partners.adjust.com/placeholders/

For example:

Then we should create events in adjust interface:


It has default events and we add some custom, Every custom event has unique token which we will give to mobile developer. Attributes should be connected to event. For example, registration event has account number and account type attributes. There is callback URL on the screen. We should put there link of callback server and list of placeholders in format key={value}. Adjust will replace value.

When mobile developers successful add SDK to the app, they can emulate activity and we can open adjust dashboard in order to see new events. Adjust has two environments. We can test, using SANDBOX in order not to mix with real data.

Going to statistics tab we can look at on default and custom events:


It is interesting fact, but install in mobile tracker it is just first launch. There is no integration with stores.


In addition we can look on events across channels or look at trends:


In order to create acquisition channel, we should go to App tab and click at "New tracker", adjsut will create link. And we can easy add to this link placeholders and custom parameters in order to collect this data in database.




Also it allows us to use cohort menu. For example, we can quickly to build retention profile:


Finally, we can look at rank and rating of mobile app. Unfortunately, we can extract this information from adjust.

The most effective tool for creating dashboard

We have wonderful dashboard in our openspace. I highly reccomend to use this cloud software for consolidation of various KPIs, SLAs and etc.

Let's meet Klipfolio you can quickly to try demo and build your own clips based on your data.

It works very easy. Let's look foundation of klipfolio.

1. Connect your data
Automated data retrieval from 1000s of data services (with more added every day).
Klipfolio is backed by a simple, flexible data architecture. That means you can upload any data source to the dashboard, automate its retrieval, and use (and reuse) that data to build visualizations and dashboards for your team.

 Forget the confusing schemas and intimidating data warehouses. From uploading your data to using it to create visualizations, there is no step in between to complicate matters (and no need to hire a team of developers to get started). Just upload your raw data to start using it to create a visualization.

 Data and presentation are treated separately in Klipfolio which was harder for us, but easier and better for you. Here's why it's better: You can use a single data source to create multiple data visualizations, or use multiple data sources to create a single visualization. What you do with your data is entirely up to you.

By default, all web accessible resources are set to automatically refresh every 30 minutes, but you can boost that up to every minute for real-time data sources. If Klipfolio detects a change in your data, your dashboard will automatically update.

Of course, you can always leverage the Klipfolio API to automate the retrieval of any other data source.







2. Visualize your data
Bring the story of your data to life with custom visualizations.

Everyone loves a good story. Stories have the ability to teach memorable lessons, change behaviour, and inspire action. With Klipfolio Dashboard, you get all the tools you need to bring the story of your performance to life with clear, compelling visualizations.
And you can bet that it's a story that will motivate your data-driven team.






Klipfolio includes 15 different types of visualizations so you can choose a visualization that will have the maximum impact on your dashboard.

 

Role of web analytics in the organization

This bundle of product can solve any marketing task and it is easy integrate with eachother. But the price is pretty expensive for the leader. Of cource, it is possible to use open source solutions and combine with cheap marketing solutions in order to create ZOO of tools but it all depends on marketing budget.

As analyst I should work with Web Analytics and I want to cover questio, why do we need web analytics tool and who should use it.

Basicaly web analytics tool can answer on questions like:

• Which website promotions are getting visitors to click and convert?
• Where are website conversions funnels breaking down and causing the organization to lose visitors?
• Which products on the website are performing the best?
• What is average time on site per visitor?
• What pages does customer visit before checkout?
• Which content or tools on the website are having the biggest impact on converting?
• and many others

Standart implementation of web analytics tools consist of 5 steps:

1. Identifying the business questions to be answered
2. Finding the best way to answer those question via collected data
3. Tagging our website
4. Development of reports and dashboards
5. Dive in reports and dashboards in order to find business insights 

We can define the following roles for using SiteCatalyst:

• Head of Digital Intelligence  - often meet with top management team and communicate with his team in order to understand how things are
• Web Analyst - work with data, invent hypotheses and check them, create segments, A/B test, reports and etc in order to produce valuable information
• Developer - guys who can work with code
• Marketers - they spend money on implementation of web analytics tool but haven't enouth time to attend training or even log on into account. In case if they successful log on they can't creat proper report.

Adobe Marketing Cloud Brief overview

As a Digital Analyts, I have to work with Adobe Marketing Cloud, it is a leader on the market.
According to Forrester Adobe is a leader

Adobe offers lots of producs for marketing. It calls adobe cloud marketing.

Analitycs - It’s the industry-leading solution for applying real-time analytics and detailed segmentation across all of your marketing channels. Use it to discover high-value audiences and power customer intelligence for your business. Here we can finde main web anaytics product Adobe SiteCatalyst. Morover it has mobile marketing services for mobile analytics.

Audience Manager - It’s a data management platform that helps you build unique audience profiles so you can identify your most valuable segments and use them across any digital channel.

Campaign - It's a set of solutions that help you personalize and deliver campaigns across all of your online and offline channels. In other words it is campaing management tool like IBM Unica.

Experience Manager - It’s a comprehensive content management solution for building websites, mobile apps and forms. And it makes it easy to manage your marketing content and assets.

Media Optimizer - It’s a programmatic ad-buying solution that helps you forecast the best mix of search, display and social ads based on your budget. It also automates the execution of your media plan and helps you find the best way to deliver relevant content to your high-value audiences.

Primetime - It’s a multiscreen TV platform that helps broadcasters, cable networks and service providers create and monetize engaging and personalized TV and film experiences.

Social - It’s the social management platform that ties every piece of data to your bottom line. Beyond just the likes and follows, manage the deep relationship between your customers’ sentiments and your business goals.

Target - It’s a personalization solution that makes it easy to identify your best content through tests that are easy to execute. So you can deliver the right experience to the right customer.

Tealium Web Page Processing Order

To understand how Tealium’s Tag Management System (TMS) loads, we should look at how a

browser loads a web page.

1. First, the browser requests the web page’s HTML (Hyper Text Markup Language) document.

2. The browser begins to download and parse the HTML document starting at the top-left and

continuing down to the bottom-right. As the browser reads through the HTML it encounters various

elements that make up the content of the web page. These elements can be any images, links,

text, or scripts like utag.js.

3. The browser continues to download these elements as it encounters them while it downloads

the HTML document itself. In the case of regular scripts, the browser must stop downloading the

HTML and interpret the script before continuing on to the rest of the HTML. Smart scripts, like

utag.js, have a built-in halt that pauses the script and allows the browser to continue downloading

the HTML. The halt is released when the browser signals that the HTML has been fully

downloaded and parsed into the Document Object Model (DOM). This signal is typically called

‘DOM Ready’ (more on this later).

4. The browser begins to render the web page on the visitor’s screen.

Order of Operation

The utag.js operations flow in a very specific order and understanding that order can prevent errors in

your implementation.

1. Run Extensions scoped to “Pre-Loader” and set Privacy Manager cookie

2. Process Data Layer

3. Evaluate Load Rules

4. Run Extensions scoped to “All Tags” and the Privacy Manager Extension

5. Load Bundled Vendor Tags and Vendor Tags with their Wait Flag set to “Off”

6. Load Vendor Tags, their Extensions, and Mappings

7. DOM Ready and the Send Function

8. After Page Load: utag.view() and Advanced Configurations

Processing of utag.js:

Step 1: Pre-Loader Extensions & Privacy Manager Cookie

Pre-Loader extensions run first. Since Pre-Loader extensions run before everything else, it is vital

that they not rely on any of the Tealium elements (data layer, other extensions, etc.) that run later.

Attempting to do so could result in the page loading improperly, or not at all. The order in which PreLoader extensions are processed is based upon their order in Tealium iQ. The Tealium Privacy cookie is set at this point, if it does not yet exist.

Step 2: The Data Layer is Built

The utag_data object, also referred to as the Universal Data Object (UDO), passes data from your

web page to Tealium iQ. The variables and data contained in utag_data are combined with data from

the rest of the web page, 1st-party cookies, and the DOM to create the utag.data object. While the

utag.data object contains all the web page’s data, only the data you identify as data sources are used

in the Tealium iQ console for mappings.

Step 3: Load Rules

The Load Rules are evaluated. The Load Rules determine if particular Tags load on any given

web page. If a Load Rule determines that a Tag should not load on a web page, that Tag and any

extensions scoped to it do not load. At this point, all of the data on the page may not be available as

the browser has not yet finished parsing. The Load Rules are re-evaluated later, when the rest of the

page’s data is available, to make certain they evaluate correctly.

Step 4: Extensions Scoped to “All Tags” & Privacy Manager

Any extensions scoped to “All Tags” run. These extensions execute in the order in which they appear

in Tealium iQ. This scope indicates that the data that come from these extensions are available to all

the Tags in your profile.

The Privacy Manager Extension, if enabled, runs with the extensions scoped to “All Tags”. The

Privacy Manager provides your visitors a way to manage their privacy settings for the various

Tags in use on your site. The visitor is provided with the option to opt in or out (depending on your

configuration) of Tags, if they so choose. It is essential the Privacy Manager runs before any other

Tag in order to abide by local restrictions or laws regarding visitors’ privacy.

The first thing the Privacy Manager does is to check for the Tealium Privacy cookie to determine the

privacy settings. Depending on your visitor’s preferences, the Privacy Manager may prevent some or

all of your Tags from loading. The Privacy Manager saves the visitor’s privacy settings in the Privacy

Manager cookie for the next time he or she visits your site.

Step 5: Bundled Tags and Tags With the Wait Flag set to “No”

Two types of Tags load at this point: bundled Tags and Tags with their Wait Flag set to “No”. Bundling Tags means that all of the Tag requests are placed into one large request The Wait Flag is an advanced setting that allows you to choose whether or not the Tag waits for the DOM Ready signal before it begins loading.

These Tags will load based on the initial evaluation of Load Rules and visitor privacy settings set by

the Privacy Manager.

Step 6: Vendor Tags, their Extensions, and Mappings:

Depending on whether or not the Load Rules permit, the vendor Tag libraries load next. If any of

those Tags have extensions scoped to them individually, those extensions load along with the Tag.

After the Tag libraries and Tag-specific extensions are loaded, the page data variables you identified

as data sources are mapped to the vendor Tags’ destination variables. As discussed earlier, even

though the Tag libraries load, they don’t do anything until the DOM Ready signal is received.

Step 7: DOM Ready and the Send Function

When the browser sends the DOM Ready signal, DOM Ready-scoped extensions and Tag-specific

extensions execute and data source mappings are processed. The Send function collects the final

vendor Tag requests together and sends them to each of the vendor services. The order in which you

have placed your Tags in Tealium iQ is the order in which the data is sent to each vendor.

Step 8: After Page Load: utag.view() and Advanced Configurations

We can use tealium web companion  (tealium tools extension for Chrome), in order to look what is load on our page.

Tealium IQ objects

I mentioned about Tealium IQ in my previous post. It is real powerful tool in age of digital transformation where speed and consolidations are vital.

Let's look inside in order to meet IQ objects.

Data Layer which has Data Sources and Bundles.

Data Layer is a collection of the data from your web page that is critical to your digital marketing campaigns all centralized in to one location in Tealium iQ.  There is an example of data layer, where we have all our objects:

By configuring data points to use in the Data Layer, such data can be shared with marketing technology tags configured inside the Tealium Tag Management platform, but can also be used to fire digital marketing technologies. Data Layer is not the Data Object; the Data Object is a JSON, a universal format which can work with all web browsers and systems alike. The Data Object forms part of the Data Layer.

We have 6 types of data you can choose from to compose your Data Layer.  

We call these data types data sources and they can be broken down in to the following categories: Universal Data Object, JavaScript Page Variable, Meta Data, Cookies, Query String Parameters and DOM Data. 

UDO – The UDO is an acronym for Universal Data Object – Your UDO is a code based representation of all the business critical information you want to know about a page when it loads.  For example, you will want to know the pages name, any information that may be related to the site like its country or language or the site name.  You may want visitor related information or to be aware of certain events like which products are being viewed or any product related ecommerce data.  Any information that may be required or useful to report you can place in your UDO.

JavaScript Page Variables – These are variables on your page that are outside of your Universal Data Object that hold a value that can be changed.  They can be text, number or true/false Boolean values, or something like a carousel click or an event for link tracking.

 Meta Data – is used when you are referencing the content on your page.  Information on your page may include things like author name of a blog or keywords or a title of an article.

Cookies – Tealium only tracks first party cookies.  You will select this data source when you are referencing a value in a cookie. 

Query String Parameters – When a data (or a variable) is passed within the URLs query string, you can point Tealium to that data and collect it in your data layer.  The querystring is anything in the URL beyond the question mark.  The querystring is set up in name value pairs that are separated by an equal sign, and each pair is separated by an ampersand.  If you have ever seen a referrer code in your querystring set to a specific value, then you have seen a querystring parameter.  The name is the referrer code and the value is what it is equal to. 

DOM Data – Any data stored in your Document Object Model, or DOM,  is automatically available to you in Tealium Data Layer and does not have to identified in your Data Layer.

AudienceStream Attributes are treated as imported data sources in Tealium iQ. This means you cannot edit or delete them from Tealium iQ. If you want remove or modify an Attribute, you must do so from your AudienceStream profile. Moreover, you cannot send Attributes to a profile library, only to a regular profile.

Bundles - list of predefined objects in order to quickly build Data Layer based on the most commonly used Data Sources and Tealium best practices. There are Standard Bundles ranging across a variety of uses from E-Commerce to Mobile Apps and Provider Bundles from such services as Demandware and Wordpress. For example Customer Data Sources:

The Tags tab gives you the ability to load your Tags on your websites. We support over 600 hundred different Tags in our Tags marketplace. Adding and configuring your Tags through Tealium iQ can be as easy as a few points and clicks. In other word we can choose any vendor from the list in order to get predefine tag. For example, on screen below I can easy deploy, google analytics, mixpanel and kissmetrics in order to send to these systems data from data layer.

Load Rules - allow me define rules for particular tag. For example, we can run tag only on landing page or during checkout.

Data Mappings - the last step during setup of tag. It allows me to map tealium iq objects with 3rd system. For example, I can collect data through data layer and send it to Google Analytics.

Extensions gives us power to manipulate with data objects in order to use various functions and build new objects. For example, we can calculate new field, or set any value to cookie.