Python Developer

How can I read the following JSON structure to spark dataframe using PySpark? My JSON structure {&#… Read more Read Json File As Pyspark Dataframe Using Pyspark?

I'm loading many versions of JSON files to spark DataFrame. some of the files holds columns A,B… Read more Spark - Set Null When Column Not Exist In Dataframe

I am having some troubles reading the following CSV data in UTF-16: FullName, FullLabel, Type TEST.… Read more How To Read Csv File With Additional Comma In Quotes Using Pyspark?

This solution in theory, works perfectly for what I need, which is to create a new copied version o… Read more How To Use To_json And From_json To Eliminate Nested Structfields In Pyspark Dataframe?

When you join two DFs with similar column names: df = df1.join(df2, df1['id'] == df2['i… Read more Removing Duplicate Columns After A Df Join In Spark

I'm looking for a way to aggregate by hour my data. I want firstly to keep only hours in my evt… Read more Hourly Aggregation In Pyspark

I have a StructField in a dataframe that is not nullable. Simple example: import pyspark.sql.functi… Read more Can I Change The Nullability Of A Column In My Spark Dataframe?

I have the following DataFrame: +----------+ | date| +----------+ |2017-01-25| |2017-01-21| |… Read more How To Convert Date To The First Day Of Month In A Pyspark Dataframe Column?