OneTick Blog

4 min read

Big Data, Big Opportunities and Challenges in Finance

Nov 19, 2018 3:34:42 PM

There has been an explosion of hype surrounding Big Data. The term Big Data has largely been associated with loosely-structured content, originating from web search companies and social media.

Social Big Data is about gleaning meaningful value from unstructured content. It is judging the mood of the human psyche, which could be Twitter, Facebook, websites commentary and blogs.  In the hunt for business benefit, any one or group of data points cannot be valued as accurate or inaccurate only factors that influence behavior.  The science of social data is the alchemy of behavioralClose up of businesswoman holding graphs in hand-1 targeting involving not only what to keep, but what to throw away.

Conversely, in capital markets the need for reliable and accurate data is the driving force. Drilling down to the individual constituent is a mandate, be it a stock symbol, option strike, currency pair or futures contract.  In the fiercely competitive trading industry it is tighter spreads, thinner margins and a lower risk appetite that evoke a wider hunt for alpha as firms look to cross border, cross asset trading models and a reenergized focus on controlling costs. This has exponentially increased demand for deep data over longer time periods across global markets; equities, futures, options and currencies.  In the Options industry, trading volume last year was four and a half billion contracts, OPRA peaked at four million messages a second. The Options market is a quintessential example of Big Data’s volume and velocity definition.

Financial Big Data is about the capture and storage of this deep data and linking disparate data sets under some common thread to tease out an intelligible answer, a diamond from a mountain of coal. It’s the quantitative research to find the cross asset correlations or understanding how to best hedge a position to offset risk.

Firms demand confidence in the resulting analytics derived from Big Data… those used in determining the profitability profile of new models, optimizing parameters through backtesting of existing models and re-balancing portfolios. For this they all depend on accurate, clean market data across all their tradable markets and the capture of order executions. This big data dump is the fuel that drives the engine of the trade life cycle from: 

  • Alpha Discovery and Research
  • Strategy Development
  • Strategy Backtesting and Optimization
  • Portfolio Valuation and VaR Management, Transaction Cost Analysis

It’s Big Data and the assurance of its accuracy, reliability and timeliness that powers this entire process, the engine of the trading industry.

The Challenge of Big Data... in Finance

Yet Big Data is messy. When it comes to financial data, market data of all stripes comes in many shapes, sizes and encodings. It continually changes and requires corrections and an occasional tweak. Market practitioner’s worst fears are spending more time processing and cleaning data than analyzing it. To focus on discovering new alpha and optimizing existing strategies, finding that diamond in the rough so-to-speak, demands a confidence in the resulting derived analytics. It means dealing with ….

  • The vagaries of multiple data sources
  • Mapping ticker symbols across a global universe  of exchanges & geographies
  • Managing dynamic (rolling) symbologies (i.e. futures contracts)
  • Ingesting cancelations and corrections to create clean prices
  • Applying corporation action price and symbol changes 
  • Detecting (missing) gaps in history, that could influence outcomes
  • Coping with exchange calendars, venue trading hours and holidays
  • Managing complex order book structures

These are huge data management obstacles to overcome not easily solved by the common storage engine.  Normalizing and cleaning data is a major obstacle to ensure accuracy.

The competitive forces driving Big Data's accuracy, reliability and timeliness are many. Within the next three years 80% of Hedge Funds will be trading algorithmically, creating an ever increasing number of firms chasing after an ever diminishing pot. Looming over the industry like a rain cloud is regulatory action, will there be order cancelation fees or minimum resting periods imposed?   These forces will drive deeper liquidity analysis across fragmented markets researching order book dynamics in that hunt for alpha.

The Prospects of Big Data... in Finance

Where are we heading with Big Data? Data volumes will continue to grow across all asset classes and firms will continue to reach deeper to weather that perfect storm of thinner trade margins, regulatory actions and cost containment.

Disparate data sources will begin to play a greater role in trade analysis. The analytics behind linking market data and social data will expand beyond news sentiment. We will begin see to firms try to understand that to their advantage.

We’ll begin to will see non-financial vendors enter the space – primarily on the analytics side. As they try to capitalize on their knowledge and skills in social analytics.

Marketplace providers and exchanges will broaden their offering to include more than market data as social content becomes more relevant in trade decision modeling.  Just as in the retail industry where user opinion/product reviews are highly valued as a decision tool for buyers, the combination of market, news and social content will drive a new style of trade model.

As the hype subsides, the Big Data definition and understanding will clarify. But unlike hardware technology which always seems subject to commoditization the business value of data will always remain and only get higher.

To learn more about OneTick and our ability to provide reliable and accurate Big Data in the Capital Markets , please visit us HERE

Louis Lovas is the Director of Solutions at OneTick where his responsibilities include strategic business development and delivering targeted solutions for quantitative research.

If you are not subscribed to our blog, please SUBSCRIBE HERE

Follow Us on Twitter and LinkedIn for more updates and information regarding OneTick.


Louis Lovas
Written by Louis Lovas

Louis Lovas is the Director of Solutions at OneMarketData where his responsibilities include strategic business development and delivering targeted solutions for quantitative research and trading systems. Louis brings over 20 years of experience in developing cutting edge solutions for financial markets and is a leading voice on industry trends, trading technology and data management for Capital Markets. Prior to joining OneMarketData, Louis was the Chief Architect and Fellow of Apama products (formerly from) Progress Software. Earlier in his career, Louis held numerous positions in the Financial Systems and Corporate Program Management divisions at Unisys Corporation (formerly Burroughs). Louis has a B.S in Business Administration/Management.

Post a Comment