2018 Big Data Trends: Liberate, Integrate & Trust Syncsort conducted its fourth annual survey of IT professionals working with Big Data to get a real-world perspective on the opportunities and challenges facing enterprises as we enter 2018. Almost 200 IT professionals, all of whom have some involvement with Big Data, participated, representing a wide range of industries. Its clear that what had been new, unfamiliar technology when we first started the survey, is now becoming an integral part of many IT landscapes. More than 40% of respondents report being in production with Hadoop or Spark, and more than 30% say they are engaged in a proof of concept or pilot program. We wanted to uncover what priorities are driving these organizations. What challenges are the toughest to overcome? How are they going about it? What data is essential? And what benefits are companies seeing in their bottom line? Read on to learn what they told us. Executive Summary What is your current role? 24.0% 20.8% 12.5% Other Data Architect Developer 15.1% IT Manager Data Scientist 11.5% 4.2% Other IT Executive 7.3% Business Intelligence/ Data Analyst 4.7% CIO CTO COO What industry is your company in? Financial Services Healthcare Other Government Retail Insurance Data Services Telecommunications Non-profit Travel & Hospitality Software or Software as a Service Information 19.8% 13.0%13.0% 8.9% 8.3% 7.3% 6.8% 4.2%4.2%4.2% 3.6%3.6% 3.1% Transportation (select all that apply) What Data Lake use cases are of most interest to you? (select all that apply) 70.8% ETL Data Lake Use Cases The distributed architectures of Hadoop and Spark are especially adept at data integration and manipulation, whether that takes the form of traditional ETL, offloading data and workloads from legacy systems, filling an active archive, or simply blending disparate data types. The insights from analytics that drive Big Data projects have the high- tech plumbing of distributed integration powering them. Advanced/ Predictive Analytics 63.5% 39.5% Data Blending Other 0.5% Active Archive 30.2% Offload Data and/or Workloads from Legacy Systems such as EDW or Mainframe 29.2% Clickstream Analysis & Social Media Data 22.9% Internet of Things 16.7% 60.4% Real-time Analytics Data Discovery & Visualization 53.1% 45.3% Operational Analytics NoSQL databases Files from Third-Party Data Providers or Partners Cloud Repositories Mainframe Web/mobile/social media AIX Power Systems Other RDBMS 69.3% Enterprise Data Warehouse 0.5% Whats Filling the Data Lake What data sources are/will be used to populate your data lake (select all that apply) A wide variety of data is filling data lakes, including data from mainframes, relational databases and enterprise data warehouses, along with relatively newer streaming and NoSQL data. In addition, as more organizations leverage cloud as a deployment platform, it is also gaining importance as a data source. Pulling all data in from across the enterprise is key to getting the benefits promised by this technology. Leaving any data source behind means a gap in knowledge, a blind spot in corporate insight. Putting ALL enterprise data together, blending and integrating it, then analyzing that holistic view provides the greatest benefits from Big Data implementations. Machines/ sensors 16.1% 62.5% 46.4% 45.3% 40.6% 31.8% 30.2% 30.2% IBM i17.7% Do you see Hadoop/Spark as a way to (select all that apply) An implementation of Big Data technology provides a wide variety of business benefits. Data Lakes are helping businesses cut costs through operational efficiencies, cost- effective storage strategies, and avoiding compliance fines. This technology is also boosting revenue by increasing productivity, extending the capabilities of existing infrastructure, increasing agility, and providing better business insights with advanced analytics. Many Benefits to the Business Increase business user productivity with more readily available data across the organization Increase operational efficiency and reduce costs 59.9% Take advantage of next- generation analytics Increase revenue and accelerate growth based on better insights Support a cost-effective storage/data archive strategy 59.4% 55.7% 54.7%55.2% 41.7% 32.8% 30.2% 21.4% Increase business/ IT agility Complement/ extend your existing large EDW or mainframe investments Facilitate regulatory compliance by retaining data long term Free-up mainframe resources and reduce costs How important is the ability to collect data across traditional and Next-Gen platforms to optimize capacity planning and management? (select all that apply) Another, often overlooked, benefit of Big Data projects is getting a better understanding of capacity.Over 90% of respondents found some capacity management value from Data Lake projects. 5.7% Somewhat ValuableValuable 51.0% 21.6% Very Valuable 20.1% Slightly Valuable 1.5%Not Valuable Please rank the challenges/barriers for implementing a Data Lake (1 = mo