SparkContext in PySpark

SparkContext in PySpark
Run this on Jupyter Notebook after installing Spark

from pyspark import SparkContext

sc = SparkContext()

%%writefile example.txt
first line
second line
third line
fourth line

textFile = sc.textFile('example.txt')

textFile.count()

textFile.first()

secfind = textFile.filter(lambda line: 'second' in line)

# RDD
secfind

    PythonRDD[7] at RDD at PythonRDD.scala:43

# Perform action on transformation
secfind.collect()

    ['second line']

# Perform action on transformation
secfind.count()

Comments

Difference between .exec() and .execPopulate() in Mongoose?

Here I answer what is the difference between .exec() and .execPopulate() in Mongoose? .exec() is used with a query while .execPopulate() is used with a document Syntax for .exec() is as follows: Model.query() . populate ( 'field' ) . exec () // returns promise . then ( function ( document ) { console . log ( document ); }); Syntax for .execPopulate() is as follows: fetchedDocument . populate ( 'field' ) . execPopulate () // returns promise . then ( function ( document ) { console . log ( document ); }); When working with individual document use .execPopulate(), for model query use .exec(). Both returns a promise. One can do without .exec() or .execPopulate() but then has to pass a callback in populate.

Machine Learning — Supervised, Unsupervised, and Reinforcement — Explanation with Example

🤖 Let's take an example of machine learning and see how it can be performed in three different ways — Supervised, Unsupervised, and Reinforcement. We want a program to be able to identify apple in pictures Supervised Learning You will create or use a model that takes a set of pictures of apple and it analyses the commonality in those pictures. Now when you show a new picture to the program, it will identify whether it has an apple or not. It can also provide details on how confident is the program about it. Unsupervised Learning In this method, you create or use a model that goes through some images and tries to group them as per the commonalities it observes such as color, shape, size, partern, etc. And now you can go through the groups and inform the program what to call them. So, you can inform the program about the group that is apple mostly. Next time you show a picture, it can tell if an apple is there or not. Reinforcement Learning Here the model you create or...

269. Alien Dictionary

Solution This article assumes you already have some confidence with graph algorithms , such as breadth-first search and depth-first searching . If you're familiar with those, but not with topological sort (the topic tag for this problem), don't panic, as you should still be able to make sense of it. It is one of the many more advanced algorithms that keen programmers tend to "invent" themselves before realizing it's already a widely known and used algorithm. There are a couple of approaches to topological sort; Kahn's Algorithm and DFS. A few things to keep in mind: The letters within a word don't tell us anything about the relative order. For example, the presence of the word kitten in the list does not tell us that the letter k is before the letter i . The input can contain words followed by their prefix, for example, abcd and then ab . These cases will never ...

Vinit Khandelwal

Search This Blog