Integrating Spark into the BI Application - The InetSoft Approach

Analytic Query Acceleration

Spark is a general purpose computing platform. Although it was designed to handle real-time computing needs, it provides no special treatment for analytic queries. A BI tool could use Spark as-is and get decent performance. But BI queries generated by interactive analysis often have distinct patterns, and it is possible to use this knowledge to further optimize the execution. As a natively integrated tool, Style Intelligence is in a unique position to add special logic to improve performance further. The acceleration is achieved by three methods:

1. StyleBI analyzes a visualization and generates candidate queries to materialize. The goal is to preprocess as much as possible and only leave the parts that need to be dynamically updated to be performed later.
2. For data stored in a materialized format, logic responsible for handling interactive queries is pushed down to the data storage layer whenever possible. This significantly reduces the amount of data Spark needs to process.
3. A specialized columnar data store is created to store the materialized data. It conforms to the standard Hadoop/Spark API so it can be accessed by Spark jobs, but is optimized to handle the types of queries generated from interactive analysis.

Spark ML

Spark ML provides an opportunity to bring Machine Learning to the masses, and we view StyleBI as the bridge from Spark ML to business users. The integration has the following objectives:

1. Creating and training models should be easy and accessible to people with minimum Machine Learning training. Every opportunity was taken to automate the process.
2. Once a model is trained, it should be available to business users with no Machine Learning knowledge.
3. Like Spark queries, Spark ML integration is done at the native level, fully integrated into the product, so it enjoys all the benefits of the cluster.

Spark ML provides built-in algorithms in the following areas:

1. Classification - prediction of categorical values
2. Regression - prediction of numeric values
3. Clustering - automatic grouping of similar items
4. Recommendation

The first three are available in StyleBI and can be accessed as a regular query without any programming or special knowledge.

Learn about the top 10 features of embedded business intelligence.

Summary

Big Data is more than throwing a cluster together and connecting to it through a JDBC driver. It requires an architecture that has all the pieces working closely together. It's our view that the most effective solution requires a complete fusion of the various technologies.

The StyleBI/Spark integration is the first step in achieving this vision. As the innovations flourish, our mission is to create a platform that effortlessly integrates with software in the Hadoop ecosystem, so business users can enjoy all the benefits.

Related Resources About Spark

• Data Grid Caching Technology

  Explains how InetSoft connects to large-scale Spark sources and uses an in-memory data grid cache to avoid repeatedly scanning huge datasets, improving dashboard responsiveness. Describes use cases where Spark-held data can be queried and blended with other enterprise sources without moving the original data. Highlights the performance benefits of caching compressed, analytic-ready data blocks for interactive visual exploration. Recommends InetSoft’s approach for organizations that need real-time style dashboards over very large Spark datasets.

• Pushed Into The Cluster

  Describes how InetSoft’s BI engine integrates natively with Spark so that analytic processing can be executed directly inside the cluster rather than extracting raw data out. Explains that in-memory tables can be pushed into the cluster and join operations executed there to preserve performance and avoid unnecessary data movement. Notes that Spark SQL and direct file connectors are supported so clusters appear as first-class analytic sources. Emphasizes the practical benefits for hybrid workloads where some transforms are best done on-cluster.

• Traditional Relational Database

  Frames Spark as a queryable data source that can, for BI purposes, be treated like a relational backend using Spark SQL and standard drivers. Discusses alternate integration strategies—connecting via JDBC/ODBC, or embedding analytic jobs into the cluster so the BI tool leverages cluster compute. Covers how Spark can replace or augment ETL by performing heavy transformations natively before BI consumption. Suggests this model simplifies handling streaming and batch data while enabling easy adoption by BI teams.

• Mash Up Such Sources

  Outlines InetSoft’s claim that its analytic engine can run natively inside Hadoop/Spark environments, allowing mashups and visual analytics to execute with full cluster power. Explains the ability to combine enterprise data from relational stores, APIs, and NoSQL with on-cluster assets without copying large volumes of data. Describes the “Big-Data-In-A-Box” option for teams that want an easy, Docker-packaged Spark-enabled deployment to avoid Big Data ops complexity. Conveys that the approach shortens time-to-value for organizations lacking deep Big Data expertise.

• Analytic Query Acceleration

  The whitepaper details how InetSoft leverages Spark to accelerate analytic query processing and scale visual analytics workloads. Covers techniques for pushing computation into the cluster, using Spark SQL, and minimizing data movement to preserve throughput and latency SLAs. Includes architectural recommendations for combining in-memory BI acceleration with Spark compute for large volumes. Offers concrete examples of performance gains and deployment patterns for enterprise BI teams.

• High Performance And Acceleration

  Describes StyleBI’s mashup engine and how it achieves high performance, including integration points with Spark for scalable processing. Explains the concept of analytic-ready data blocks and optional compressed cache blocks that deliver faster interactive performance over big datasets. Notes that combining InetSoft’s engine with Spark cluster scalability often yields substantial speedups compared with naive direct queries. Positions the product as suitable for organizations that need both self-service analytics and Big Data scale.

• Built-In Dedicated Spark

  Highlights InetSoft’s easy path to big data analytics, including an option to deploy a built-in dedicated Spark/Hadoop cluster for customers that do not yet have cluster infrastructure. Explains how the platform can either attach to existing clusters or provide pre-packaged cluster nodes (Big-Data-In-A-Box) for rapid deployment. Describes on-the-fly data transformation and mashup capabilities that allow analysts to combine diverse sources into a unified analytic environment. Advises this route for teams seeking a cloud-ready, scalable BI solution with minimal Big Data operational overhead.