UC San Diego Big Data Analytics with Spark

  so I wrote this little widget to help you visualize it. So here is the same graph, and what we have here is basically  the approximation of the function  that we can play with, we can change from,  we can change it using these sliders.  So right now, we have the approximation  that is zero everywhere,  so it's basically the zero function,  and it's clearly very far from the,  from our target function, x minus four absolute value.  So let's see what happens if we increase this coefficient.  We see that we move,  we move the constant to the middle of the function,  more or less.  Then when we add the second one,  we add a sinusoid,  and the sinusoid helps us get closer to the function,  in some of the regions.  It doesn't help us at all here,  right, because it's zero,  and it stays equidistance,  but now if we add the cosine,  that helps us also there.  So what you see her...

 It is important when you do statistical analysis that you find that part of the data which is dense. If you do not have a large number of records for a value that want to calculate such as variance or expectation you would get a value that is unreliable. DataFrames are more flexible than RDD as you can add new columns to them. For Azure HD Insights the actual HDFS storage is at the Storage account associated with the cluster at creation time. So you can directly read a parquet file from Storage account container into the cluster as below:- parquet_name='wasb:///weather/weather' query="""SELECT station,measurement,year  FROM parquet.`%s.parquet`  WHERE measurement=\"PRCP\" """%parquet_name print(query) df2 = sqlContext.sql(query) #print 'number of rows=',df2.count() df2.show(5) to_Pandas is not longer correct you should use toPandas() to convert a spark data frame to a pandas dataframe We convert a spark data frame to a Pandas dat...

 DataFrames are special type of RDD's. DataFrames have two dimensional data like spreadsheet and they have rows and they have columns. SQL context is like SparkContext and lies underneath sparkContext and handles data frames. df.printSchema defines the schema of the dataFram which is name and type Take a RDD, transform it into a dataframe and then load it into a parquet file. Variants of join. There are four variants of  join  which differ in how they treat keys that appear in one dataset but not the other. join  is an  inner  join which means that keys that appear only in one dataset are eliminated. leftOuterJoin  keeps all keys from the left dataset even if they don't appear in the right dataset. The result of leftOuterJoin in our example will contain the keys  John, Jill, Kate rightOuterJoin  keeps all keys from the right dataset even if they don't appear in the left dataset. The result of leftOuterJoin in our example will contain the keys...

 Spark context is a way of communicating with the spark system. You can only have spark context per system as spark is designed as a single user system. to stop a spark context sc.stop() RDD(Resilient Distributed DataSet) is the novel main data structure in spark. You can think of it as a list whose elements are stored in different computers and they have a component which is residing in the driver node. Once you take something and define it as an RDD it is actually going to take it longer for you to be able to read it as it is no longer available locally and it will cost you to bring this data back. The simplest way to create an RDD is to take a list and then call sc.parallelize. Collect is the reverse of parallelize and will collect the elements of the RDD in head node. Collect eliminates the benefits of parallelism. Map- Applies the given operation to each element of the RDD Reduce- Map the RDD to a single value using a given operation. Usaually the recommendation is one worker ...