My first incorta materialized view
I found that incorta allows me to practice my PySpark skills. This is my first MV.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
%pyspark | |
from pyspark.sql import SparkSession | |
from pyspark.sql.types import ArrayType, StructField, StructType, StringType, IntegerType | |
data = [('Category A', 100, "This is category A"), | |
('Category B', 120, "This is category B"), | |
('Category C', 150, "This is category C")] | |
# Create a schema for the dataframe | |
schema = StructType([ | |
StructField('Category', StringType(), True), | |
StructField('Count', IntegerType(), True), | |
StructField('Description', StringType(), True) | |
]) | |
# Convert list to RDD | |
rdd = spark.sparkContext.parallelize(data) | |
# Create data frame | |
df = spark.createDataFrame(rdd,schema) | |
save(df) |
Comments
Post a Comment