Big Data - Revision history

Christoph: /* See also */

2017-03-23T19:21:21Z

‎See also

Christoph: /* Apache Spark */

2017-03-23T19:21:03Z

‎Apache Spark

Christoph: /* Analyzing Big Data */

2017-03-14T18:59:10Z

‎Analyzing Big Data

Christoph: /* Analyzing Big Data */

2017-03-14T18:58:38Z

‎Analyzing Big Data

Christoph: /* Apache Spark */

2017-03-14T18:12:18Z

‎Apache Spark

Christoph: /* Apache Spark */

2017-03-14T17:44:45Z

‎Apache Spark

Christoph: /* Apache Spark */

2017-03-14T17:42:21Z

‎Apache Spark

Christoph: /* Apache Spark */

2017-03-14T16:18:13Z

‎Apache Spark

Christoph: /* Overview of NoSQL */

2017-03-13T23:25:08Z

‎Overview of NoSQL

Christoph: /* Overview of NoSQL */

2017-03-13T23:02:40Z

‎Overview of NoSQL

@@ Line 705: / Line 705: @@
 * [[:Category:AWS|Amazon Web Services]] (AWS)
 * [[Google Cloud Platform]] (GCP)
 ==References==

@@ Line 621: / Line 621: @@
 ===Apache Spark===
+ see: [[Apache Spark]]
-;What is Spark?
-* Fast and general engine for large-scale data processing and analysis
-* Parallel distributed processing on commodity hardware
-* Easy to use
-* A comprehensive, unified framework for Big Data analytics
-* Open source and a top-level Apache project
-−
-;Spark use cases
-* Big Data use case like intrusion detection, product recommendations, estimating financial risks, detecting genomic associations to a disease, etc. require analysis on large-scale data.
-* In depth analysis requires a combination of tools like SQL, statistics, and machine learning to gain meaningful insights from data.
-* Historical choices like R and Octave operate only on single node machines and are, therefore, not suitable for large volume data sets.
-* Spark allows rich programming APIs like SQL, machine learning, graph processing, etc. to run on clusters of computers to achieve large-scale data processing and analysis.
-−
-;Spark and distributed processing
-* Challenges of distributed processing:
-** Distributed programming is much more complex than single node programming
-** Data must be partitioned across servers, increasing the latency if data has to be shared between servers over the network
-** Chances of failure increases with the increase in the number of servers
-* Spark makes distributed processing easy:
-** Provides a distributed and parallel processing framework
-** Provides scalability
-** Provides fault-tolerance
-** Provides a programming paradigm that makes it easy to write code in a parallel manner
-−
-;Spark and its speed
-* Lightning fast speeds due to in-memory caching and a DAG-based processing engine.
-* 100 times faster than Hadoop's MapReduce for in-memory computations and 10 time faster for on-disk.
-* Well suited for iterative algorithms in machine learning
-* Fast, real-time response to user queries on large in-memory data sets.
-* Low latency data analysis applied to processing live data streams
-−
-;Spark is easy to use
-* General purpose programming model using expressive languages like Scala, Python, and Java.
-* Existing libraries and API makes it easy to write programs combining batch, streaming, interactive machine learning and complex queries in a single application.
-* An interactive shell is available for Python and Scala
-* Built for performance and reliability, written in Scala and runs on top of JVM.
-* Operational and debugging tools from the Java stack are available for programmers.
-−
-;Spark components
-* Spark SQL
-* Spark Streaming
-* MLib (machine learning)
-* GraphX (graph)
-−
-;Spark is a comprehensive unified framework for Big Data analytics
-* Collapses the data science pipeline by allowing pre-processing of data to model evaluation in one single system.
-* Spark provides an API for data munging, ETL, machine learning, graph processing, streaming, interactive, and batch processing. Can replace several SQL, streaming, and complex analytics systems with one unified environment.
-* Simplifies application development, deployment, and maintenance.
-* Strong integration with a variety of tools in the Hadoop ecosystem.
-* Can read and write to different data formats and data sources, including HDFS, Cassandra, S3, and HBase.
-−
-;Spark is ''not'' a data storage system
-* Spark is ''not'' a data store but is versatile in reading from and writing to a variety of data sources.
-* Can access traditional business intelligence (BI) tools using a server mode that provides standard [[:wikipedia:Java Database Connectivity|Java Database Connectivity]] (JDBC) and [[:wikipedia:Open Database Connectivity|Open Database Connectivity]] (ODBC).
-* The DataFrame API provides a pluggable mechanism to access structured data using Spark SQL.
-* Its API provides tight optimization integration, thereby enhances the speed of the Spark jobs that process vast amounts of data.
-−
-;History of Spark
-* Originated as a research project in 2009 at UC Berkeley AMPLab.
-* Motivated by MapReduce and the need to apply machine learning in a scalable fashion.
-* Open sourced in 2010 and transferred to Apache in 2013.
-* A top-level Apache project, as of 2017.
-* "Apache Spark is the Taylor Swift of big data software. The open source technology has been around and popular for a few years. But 2015 was the year Spark went from an ascendant technology to a bona fide superstar".<ref>[http://fortune.com/2015/09/25/apache-spark-survey/ Survey shows huge popularity spike for Apache Spark]. Fortune.com. 2015-09-25.</ref>
 ===Analyzing Big Data===

@@ Line 752: / Line 752: @@
-   | Analytics   | Human input | Decision  | Action
+    | Analytics   | Human input | Decision  | Action
----+-------------+-------------+-----------+----------------------+
+ ---+-------------+-------------+-----------+----------------------+
- D | Descriptive                                                  |
+  D | Descriptive                                                  |
- A | Diagnostic ---->                                             |
+  A | Diagnostic ---->                                             |
- T | Predictive ----------->                                      |
+  T | Predictive ----------->                                      |
- A | Prescriptive --------------> Decision support                |
+  A | Prescriptive --------------> Decision support                |
-   | Prescriptive --------------------------> Decision automation |
+    | Prescriptive --------------------------> Decision automation |
----+-------------+-------------+-----------+----------------------+
+ ---+-------------+-------------+-----------+----------------------+
 ==Glossary==

@@ Line 708: / Line 708: @@
 *** Visualizations (e.g., Tableau, Qlik)
 *** Programming languages (e.g., R, D3.js)
 * '''Predictive''':
 ** What could happen in the future?
 ** An analysis of historical data to predict what might happen. Yields a forecast of a probable outcome.
+** Analyzes historical data to detect patterns and trends and make predictions on probable future outcomes.
-* '''Prescriptive''':
+** Provides estimates about the likelihood of a future outcome, based on statistics and probabilities.
 ** What to do to respond to a certain future outcome?
 ** This type of analysis reveals what actions should be taken. This is the most valuable kind of analysis and usually results in rules and recommendations for next steps. This is also the most difficult type of analysis.
 ==Glossary==

@@ Line 685: / Line 685: @@
 * A top-level Apache project, as of 2017.
 * "Apache Spark is the Taylor Swift of big data software. The open source technology has been around and popular for a few years. But 2015 was the year Spark went from an ascendant technology to a bona fide superstar".<ref>[http://fortune.com/2015/09/25/apache-spark-survey/ Survey shows huge popularity spike for Apache Spark]. Fortune.com. 2015-09-25.</ref>
 ==Glossary==

@@ Line 591: / Line 591: @@
 ** Each column family can be compared to a container of rows in an RDBMS table, where the key identifies the row and the row consists of multiple columns. The difference is that various rows do not have to have the same columns, and columns can be added to any row at any time without having to add it to other rows.
 ** When a column consists of a map of columns, then there is a super-column. A super-column consists of a name and a value, which is a map of columns. Think of a super-column as a container of columns.
-** Examples include Google BigTable and Apache Cassandra
+** Examples include Google BigTable, Apache Cassandra, Hypertable, HBase, etc.
 * Graph
 ** Base on graph theory. Uses nodes, edges, and properties on both nodes and edges.
@@ Line 619: / Line 619: @@
 * [http://sungsoo.github.io/2014/05/19/column-family-stores.html Column-Family Stores]
 * [http://searchdatamanagement.techtarget.com/feature/Key-criteria-for-choosing-different-types-of-NoSQL-databases Types of NoSQL databases and key criteria for choosing them]
 ==Glossary==

@@ Line 566: / Line 566: @@
 ** Eventually Consistent Reads (default): Maximizes your read throughput. However, an eventually consistent read might not reflect the results of a recently completed write. Consistency across all copies of a data is usually reached within a second. Repeating a read after a short time should return the updated data.
 ** Strongly Consistent Reads: Returns a result that reflects all writes that received a successful response prior to the read.
 ==Glossary==