How We Built Our Own BI: When Off-The-Shelf Apps Won't Do

InformationWeek 2016 Salary Survey: 10 Best-Paying IT Leadership Jobs

In our industry, a sturdy business intelligence solution is key. It helps us power our data-driven platforms, analyze valuable consumer insights, and track our performance -- all in near real-time. This last bit is especially important, considering that at ironSource -- an online software distribution and monetization company with a complete ecosystem for downloadable applications -- we process billions of data points monthly.

Our customer success managers -- who, in a way, were our first customers -- wanted to see how campaigns were performing in real-time, so they could easily adjust and optimize them on the fly.

Naturally, we looked into several major BI providers, hoping one of them would be able to address our strategic needs. BIScience, GoodData, and Qlik were among those we evaluated.

None seemed to quite get the job done. There were a few critical metrics and KPIs missing from each.

[Don't do it like this. Read 8 Ways to Fail at DevOps.]

So, we turned instead to building an additional layer on top of one of the existing BI platforms, and then adjusting it so it could support the essential business units and features we needed.

What We Did

It took a while (six months to be exact) until we finally developed the perfect BI solution for us. Because we built it ourselves, we were able to add useful and industry-specific features. More on that later.

First, let's talk about how, during the course of this project, our small BI team acted as salespeople, customer support, Q/A, and -- of course -- developers. At ironSource, it's typical for our teams to wear many hats, so we were prepared.

In the beginning, our BI team presented our customer success managers with a trial version and asked them to use it and offer feedback. It was a struggle at first to get the customer success managers to adopt the new tool.

It was an amusing role reversal -- usually they're the ones nudging external leads to use ironSource solutions. This time it was up to us, a group of developers, to get our internal clients to integrate our portal into their everyday routines.

We sent constant email reminders with functionality updates, animations, funny gifs -- anything to catch their attention. We even enlisted the help of our in-house studio to design appealing graphics.

Truthfully, what really pushed the customer success managers to adopt the platform was showing them that our upper management team was using it, too. Our leadership demonstrated how important and necessary it was for them to start utilizing our BI portal.

Specifically, I'd send screen shots pulling numbers and statistics from the BI platform to the customer success managers and ask them to check up on certain campaigns. This way, they'd need to go into the portal in order to respond to and access the data.

This helped us create a shared language for discussing performance, results, and expectations with clients, which can be key to both successful client management and company-wide unification. Since ironSource is made up of many divisions, this is a critical challenge for us.

Customer Support

Failure was a given. But as the saying goes, if at first you don't succeed, try, try again. In the beginning, as with most new platforms, there were discrepancies between the data our portal was presenting, and the actual data. Really, the only way we could fix this particular mistake was with the help of the customer success managers, who consistently flooded our email inbox with bugs and requests.

In fact, within a few days, we set up a WhatsApp group for support, so users could direct all their concerns to it, instead of emailing people individually. Customer support was critical. If it was 3:00 a.m., my team was up and ready to fix bugs ASAP. After all, ad campaigns don't sleep. If something was off with our numbers, it couldn't wait until morning. It wasn't uncommon for our team to stay in the office until midnight during those days.

Quality Assurance

Unsurprisingly, our initial version needed quite a bit of work. In the WhatsApp group, the customer success managers outlined three major flaws:

First we tackled the UI, dividing the various different dashboards into logical groups and adding intuitive icons. Our design studio was a great help here, working with us on logos, fonts, and general design. They're UI experts, so, luckily for us, they took on most of the burden on this front.

Next, we tweaked the response time. Our platform holds millions of data points, so it needs to work fast, otherwise it becomes unhelpful, and the customer success managers fall back on using spreadsheets.

Last, we enlisted the help of ironSource's analytics team. Its members helped us calculate the complicated metrics our customer success managers requested. For them we developed hourly and monthly campaign tracking.

Sometimes customer success managers need to create monthly reports to demonstrate progress over time. To accomplish the task, we used Pentaho for ETL, which allowed us to aggregate millions of "by the minute" records into "by the month" ones.

In addition, because of our volume, we knew we needed to compare hour-to-hour results. If there's a 30% drop within an hour, the solution alerts the team so we can react quickly.

Calibrating the alerting is a fine art. On one hand, we don't want to send too many alerts (and cause alert blindness). On the other hand, we don't want to send too few (and risk missing important events). In fact, the KPIs and features we've added to our BI solution for internal use are now being used many of our external customers as well.

Of course, this is an ongoing process. We develop a new version once every two weeks based on feature requests by our own customer success managers and our external customers. Or, if a dashboard isn't being used, we remove it, try to understand why it was underutilized, and learn from it.

Where Does IT Fit in?

Our IT team was in charge of preparing the information security architecture for our BI portal. At an early stage, we had to decide if we wanted the BI portal to be available outside the office. It made the most sense to do so. After all, our customer success managers need to have access from home, because data is in real-time.

In addition, once we extended it to our external customers, they needed to be able to access the portal from all parts of the world, and in all time zones. Of course, with this decision came many security concerns.

The IT team developed a secure system to allow external access to the portal. Once it was built, they ran stress tests, vulnerability tests, and penetration tests to make sure absolutely nothing could attack and harm the system.

In addition to security management, the IT team was vital in designing and assessing the hardware requirements of the machine that runs the portal's servers.

Results

In time, we became a well-oiled machine, consistently receiving feedback from the customer success managers, assistance for implementation from the analytics team, and security maintenance reports from the IT team.

Usage of the BI platform jumped from 200 visits a day to 1,500 in only six months. (There are about 250 users on the BI platform from the business, finance, and R&D departments at ironSource.) Today, the average customer success manager accesses the platform 15 times a day, even on weekends.

In the end, building our own BI solution was perhaps the best decision we could have made. Not only did it benefit us and improve the way we tracked our own performance, it also gave rise to a new solution for our external customers. This way, they're able to use the same high-level measurements and metrics we're using, all while being able to easily interpret and understand complicated numbers on an easy-to-use dashboard.

What's your BI story? Have you attempted to build your own? Are off-the-shelf solutions meeting your needs? What role is IT playing in your BI efforts? Does adoption of a new BI solution always require a push by leadership? Tell me all about your experiences in the comments section below.