Apache Spark

Distributed data and machine learning

Apache Spark is a community-developed engine for distributing data computation across machines. Its MLlib component adds machine-learning algorithms and workflow tools, connecting data preparation to tasks such as classification, regression and clustering. That contribution concerns the processing and training pipeline rather than a model service trained and hosted by Spark itself. [2]

Version 1.0's May 2014 release established stable core APIs and expanded the surrounding libraries. It introduced Spark SQL while adding MLlib and streaming capabilities; core API stability should not be confused with a guarantee that the then-new SQL component was already final. [1][3]

The Apache Software Foundation supplied institutional and release stewardship, while Spark's project community produced the technical work. Those are complementary contributions, not evidence of a corporate parent and subsidiary. Current MLlib interfaces have also evolved since version 1.0, so today's API documentation is not a feature list for the historical release. [2][3]