Fast Data vs. Big Data - Is it the classic case of David vs. Goliath?

Here is an ebook that describes various fast data use-cases that can be addressed using an in-memory system like VoltDB.

I have been actively involved in Big Data, architecting and implementing various open-source projects in the Hadoop ecosystem, since the year 2010. Over the last few months, working with businesses that chose to address their fast data needs, I have experienced at first hand the transition from batch systems to real-time systems. It is great to be fast, but to derive an insight from piles of data is a very hard problem to solve. For businesses, how fast you act on a customer’s data to make a contextually relevant decision is as important as an insight derived from analyzing Big Data.

So as the title suggests - are the fast data and big data ecosystems competing with each other, as did David vs. Goliath, where for one to succeed the other must fail? I don't believe so. Let me try to convince you, by the end of this post, that the interplay between these two distinct ecosystems is not only mutually beneficial but also necessary.

“When I am getting ready to reason with a man, I spend one-third of my time thinking about myself and what I am going to say and two-thirds about him and what he is going to say.” 

               -  Abraham Lincoln

 In the year 2010, Apache Hadoop (which had its roots in Nutch and a few white-papers from Google) was barely four years old; only a handful of Web 2.0 companies had adopted it to index the whole web and make it searchable. In the year 2010, the original team that developed Hadoop was still part of Yahoo, and Cloudera, a young startup, was trying to compete with Amazon and still figuring out what its product should be. In the intervening years, I have seen Hadoop evolve into a much wider platform where businesses ranging from life sciences to financial services to manufacturing and many more sectors have modernized their infrastructures to realize its OpEx/CapEx benefits.

 The global Big Data market is projected to reach $122B in revenue by 2020. By 2020, every person online will create roughly 1.7 megabytes of new data every second of every day, and that’s on top of the 44 zettabytes (or 44 trillion gigabytes) of data that will exist in the digital universe by that time. In a recent Computerworld article, University of California Berkeley Professor Vincent Kelly predicted that more than a trillion devices would be in our world within the next 10 years. Frost & Sullivan forecasts global data traffic will cross 100 Zettabytes annually by 2025. As the physical and digital worlds converge, it’s nothing short of the datafication of life. It will continue to grow at record rates and will overwhelm our ability to manage effectively this amount of data. With a trillion devices on the way - the volume, velocity and variety of data is going to grow dramatically.

 Yes, organizations are amassing larger and larger datasets from a growing number of sources. However, the ability to process this data has not kept pace. It’s a massive business opportunity but also a significant potential threat to businesses. In an increasingly data intensive world your business is only as fast and effective as your competitor's ability to process its data to give real-time actionable insights. Here’s where the concept of “fast data” steps up to the plate.

Competitive edge - Fast Data gives your business wings

 Modern consumers of online content have become used to personalized product recommendations, personalized deals and even personalized communications. Companies like Amazon, Netflix and Ebay have paved the way for the modern web experience where customers are not only accustomed to tailored content but also expect hyper-personalization to get the best experience.

Hyper-personalization is the name of the game for businesses in which data plays the key role to provide a personalized and targeted service along with tailored content to enhance customer experience. Being able to act on a customer’s data in real-time and make contextually-aware decisions is very valuable to businesses. Businesses should use these opportunities to consistently target the right audience throughout the customer lifecycle. It not only aids in giving the customer a better experience, but also enables new business models such as pay as you go, which are becoming crucial in the current on-demand sharing economy.

 Fast data or data in motion is very different than big data or data at rest. Fast data applications analyze and make decisions on data as it is received in real time to have the single biggest operational impact and benefit. Mike Gualtieri, a principal analyst at Forrester Research, reports that the value of fast data is perishable. It exists in the moment and can have exponentially more value than data at rest, whereas the value of big data increases with time. Processing data at breakneck speeds requires these two technologies to work in sync with each other: a fast data system that can handle developments as quickly as they appear (preferably in a transactional fashion), and a data warehouse capable of analyzing data periodically to derive insights.

 A fast data system can support real-time analytics and complex decision-making in real time, while processing a relentless incoming data feed. As complicated as this system seems, it’s an absolute must for anyone looking to compete, particularly in the enterprise space. An example has been cited by VoltDB partner Emagine International’s CEO David Peters, who says “Subscribers receiving tailored real-time offers bought 253% more services.”

 Fast Data systems operationalize the learning and insights that companies derive from “big data”. Smarter customer engagement with real time, context-aware applications have direct impact on business results. Businesses like Emagine tap Fast Data to provide personalized content to its customers, enabling those users to gain competitive advantage through more efficient and agile operations.

 Changing economics of Data

 Search has changed our behavior. It has made the once-arduous process of finding and retaining information fast and fluid. As we try new terms, we stumble upon unexpected ideas and perspectives, which may take us off in new directions. There are no barriers to entry and no rules. Think of the economics of data the same way. It changes not only the scope and volume of data we can process, but also the way in which businesses think about data in the first place. Hadoop redefined the economics of big data, reducing the per-terabyte cost by more than 95%. When cost becomes a non-issue, adoption always soars. Smart organizations are using these new economic rules to make big data services available to anyone who needs them - but big data is only as useful as its rate of analysis. Otherwise, businesses won’t gain access to the real-time suggestions and statistics necessary to make informed decisions with better outcomes. When fast data is combined with big data, information becomes more plentiful, more actionable and more beneficial to an organization. Fast data is the way forward to operationalize the insights we get from analyzing terabytes of data and providing a better user experience.

 Fortunately, the falling cost of DRAM – approximately 30% every 12 months – has made using fast data an affordable option for just about any organization. Modern fast data products, unlike methods that may be classified as forms of supercomputing, are distinct in that they can be run on commodity hardware. More enterprises are investing in fast data using In-memory computing, and the market continues to produce an expanding range of solutions. In fact, a recent survey within the financial services industry revealed that 58% of firms use in-memory technologies for real-time analytic applications, and 28% reported that they are used in a mission-critical capacity.

 Overcoming cultural resistance

 Most companies that have big data installations in production realize the idea of big data is less about size and more about introducing fundamentally new business processes to decision making. This requires information within companies that has traditionally been siloed to be available across functional organizations. Typically this results in an uphill political battle that organizations have to constantly fight to re-architect business processes and make sure data governance/security mechanisms in place are robust. Information in a data lake is often buried in detailed unstructured data logs, and various ETL techniques are needed to extract insights from it. The value derived from the analytics piece can greatly exceed the cost of the infrastructure. To leverage big data, it is vital to have talented data engineers, data scientists, statisticians, project managers and product specialists working in tandem.

 The ROI from big data deployments of various companies across domains is evident, but it takes a lot of effort and time to materialize as it requires a cultural shift in the way companies operate. In spite of concerted efforts, most early adopters of big data have realized only 15% of the ROI from their big data installations. Analyst firm Gartner has reported, “Through 2018, 90% of the the deployed data lakes will be useless as they are overwhelmed with information assets captured for uncertain use cases.” Fast data, unlike big data which requires highly specialized skills such as R and data science, is much more agile and typically requires commodity skills such as SQL and Java. This ensures very quick turn around time for business and enables rapid prototyping of new features in products, which plays well with the fail fast engineering culture. Businesses typically realize revenue rather quickly from their fast data investments and also are to react quickly to competition, as it requires lesser effort to operationalize changes to the platform. 

Most people in the business world are at least somewhat familiar with the concept of Big Data, yet relatively few are familiar with the term ‘fast data’. I am seeing many companies across the board, ranging from Fintech to IOT/manufacturing to Ad-tech, beginning to innovate with and leverage the potential of fast data. It is tough not to draw parallels with the early days of big data, as there is a lot of excitement around fast data and in-memory computing. Perhaps the reason fast data has taken a back seat to big data in mainstream conversation is because the two are so conceptually intertwined that it’s assumed one follows the other. Businesses are rushing to allocate massive budgets to drive insights from big data without a proper strategy to operationalize these insights. I believe that eventually all businesses will compete on their ability to make decisions "in the moment" using Fast Data.

 Here are the five steps I propose as part of your fast data strategy:

1. Identify your fast data opportunity
2. Assess and leverage your existing infrastructure
3. Understand the (business implications of) alternatives
4. Get agreement on success criteria for project
5. Prototype, pilot, refine

 Here is link to a detailed guide to walk you through the process.

So, despite what the legend says, in this case, David and Goliath get to live happily ever after in their own worlds, periodically interacting as necessary. Feel free to express your opinion in the comments section or drop a line to me and I would be happy to chat.

Goal Setting Frameworks and why are they important?

I’ve been having lots of conversations with various accomplished management professionals about setting goals. Its a really important topic for many founders and aspiring professionals, because it forms the basis for management teams to align incentives and focus an organization on a few important areas. Its their focus that enables startups to move quickly, one of their key competitive advantages in the market. But, what is the optimal way of setting goals? I first learned about goal setting while I was reading "Work Rules!" by Laszlo Bock, which employs the OKR (Objective and Key Result) that Andy Grove developed at Intel. He wrote about it in a book called High Output Management. At its core, an OKR is a description of the goal with a quantitative metric of success.

Often, this type of quarterly goal setting engenders fears of linear thinking and linear progression, which is why many companies encourage their employees to aim to achieve only about 70% of their goals, and many managers at Google, demand that their employees each quarter create a moonshot goal. The likelihood of success of this aspiration is quite low, but if successful, it can meaningfully alter the trajectory of the company.

But, there are other frameworks for setting goals. I was chatting with a head of human resources at a fast-growing startup, and he described three different kinds of goals: absolute goals, relative goals, and sustainment goals. Absolute goals are things like revenue, users, employees – numbers that in a fast-growing company are monotonically increasing. Relative goals are things like market share or app rankings. Startup can aim to be in the top 10 mobile apps in a category for example. Sustainment goals are things like 99.999% uptime or 90% customer satisfaction.

This executive favored a mix across the three for every individual, because it can be challenging to be motivated exclusively by sustainment goals, for example.

Reading through some organizational behavior research, I came across another way of thinking about goals. Elliott and McGregor, two researchers known in the field, categorize goals in three different buckets. Mastery, performance approach, and performance avoidance. Mastery goals are things like becoming fluent in Spanish or in Ruby. They are unique in that the measurement of success is intrinsic, it is up to the individual to compete with themselves to achieve maximum success.

Performance approach goals are the most common. Close $1 million new business this year. Generate 1000 new leads this month. Hire six more engineers. Performance approach goals require extrinsic validation, in the form of compensation or bonuses or similar. They work well.

The last category according to behavioral research is performance avoidance goals, which asks people to demonstrate that they are not incompetent. Studies of repeatedly shown that these are detrimental and ineffective.

And then, there is a host of research on the negative aspects of goals. This Harvard paper outlines all the pitfalls of goal setting: narrow mindedness, learning inhibition, and calibration among others. For example, the Pinto, Ford’s disastrous compact car of the 70s, was a byproduct of focusing on shipping fast, and not paying attention to safety.

I have been trying to find some consistent framework for setting goals but there doesn’t seem to be an optimal mechanism of goal setting. The research is still all over the place, and different companies employ different techniques to achieve great success. Ultimately, logic and clear thinking are probably the best tools for setting goals, and motivating an organization properly.

Unikernel meets Docker - what is beyond Containers?

Docker today announced the acquisition of Unikernel Systems, a Cambridge, UK-based startup that aims to bring unikernels to the masses (or at least the masses of developers). Docker plans to integrate support for unikernels into its own tools and services as it’s starting to look at technologies beyond containers to help developers build even more efficient microservices and serverless architectures.

The basic idea behind unikernels is to strip down the operating system to the absolute minimum so it can run a very specific application. Nothing more, nothing less. This means you would compile the necessary libraries to run an application right into the kernel of the operating system.

The result of this is a very small and fast machine that has fewer security issues than traditional operating systems (because you strip out so much from the operating system, the attack surface becomes very small, too). Because of this, unikernels are great for applications where security, agility, and efficiency are paramount like microservices and serverless.