Extract Json Data From A Spark Dataframe

+------------------------------------------------------------------+ | message | +-----------------------------------------

Solution 1:

pault was right it is pretty much the same question, but you can use the following sample to achieve your dataframe output:

df_new = spark.createDataFrame([
(str({"name":"east-desktop","viewers":447,"emptyCount":0,"version":0.3}))
],StringType())

schema = StructType(
    [
        StructField('name', StringType(), True),
        StructField('viewers', IntegerType(), True),
        StructField('emptyCount', IntegerType(), True),
        StructField('version', FloatType(), True)
   ]
)
df_new.withColumn("data", from_json("value",schema)).select("value", col('data.*')).show(truncate=False)

Output:

+-------------------------------------------------------------------------+------------+-------+----------+-------+
|value                                                                    |name        |viewers|emptyCount|version|
+-------------------------------------------------------------------------+------------+-------+----------+-------+
|{'emptyCount': 0, 'version': 0.3, 'name': 'east-desktop', 'viewers': 447}|east-desktop|447    |0         |0.3    |
+-------------------------------------------------------------------------+------------+-------+----------+-------+