Python Developer

I use this as document suggests http://spark.apache.org/docs/1.1.1/submitting-applications.html sp… Read more Pyspark --py-files Doesn't Work

I'm currently taking a Big Data Class, and one of my projects is to run my Mapper/Reducer on a … Read more How To Run A Mrjob In A Local Hadoop Cluster With Hadoop Streaming?

I'm trying to do a Spark tutorial that comes with the Cloudera Virtual Machine. But even though… Read more Exceptions When Reading Tutorial Csv File In The Cloudera Vm

What is the best way to create/write/update a file in remote HDFS from local python script? I am a… Read more Python Write To Hdfs File

I followed this article to send some data to AWS ES, and I used the jar elasticsearch-hadoop. Here … Read more Write Data From Pyspark To Elasticsearch

I am trying to use subprocess.popen to run commands on my machine. This is what I have so far cmdve… Read more Subprocess Popen To Run Commands (hdfs/hadoop)

We have a hive warehouse, and wanted to use spark for various tasks (mainly classification). At tim… Read more Reading And Writing From Hive Tables With Spark After Aggregation

I am trying to do what should be a simple task: I need to convert a text file to upper case using H… Read more Hadoop Streaming With Python: Keeping Track Of Line Numbers