Big Data > Spark > What is Spark?

What is Spark?

Apache Spark is an open-source data processing engine to store and process data in the real time across various clusters of computers using simple processing constructs. It is a powerful cluster computing platform in the Big Data ecosystem, designed to be both fast and general-purpose. It extends the popular Hadoop MapReduce model by efficiently supporting a broader range of computations, including interactive queries and stream processing. Unlike Hadoop, Spark can perform computations in memory, which makes it faster and more efficient.

On the generality side, Spark is designed to cover a wide range of workloads that previously required separate distributed systems, including batch applications, iterative algorithms, interactive queries, and streaming. By supporting these workloads in the same engine, Spark makes it easy and inexpensive to combine different processing types, which is often necessary in production data analysis pipelines. In addition, it reduces the management burden of maintaining separate tools.

Spark is designed to be highly accessible, offering simple APIs in R, Python, Java, Scala, and SQL, and rich built-in libraries. So that developers and data scientists can incorporate Spark into their applications to rapidly query analyze, and transform data at scale.

              

 

No More

Next

History of Spark

Feedback

Submit