Scala> val qualifiedDeltaHdfsPath = hdfsDeltaPathPrefix + "/events" HdfsDeltaPathPrefix: String = /user/myuserid/delta Scala> val hdfsDeltaPathPrefix = "/user/myuserid/delta" |name|database|description|tableType|isTemporary| |default|default database|/user/myuserid/warehouse| Scala> val event_df = sample_df.withColumn("eventdate", to_date(col("eventdate"), "yyyy-MM-dd"))Įvent_df: .DataFrame = Scala> val sample_df = sc.parallelize(Seq(Ģ1/05/23 22:20:39 WARN SharedState: Not allowing to set or in SparkSession's options, it should be set statically for cross-session usages Scala> val columns=Array("col1", "eventdate")Ĭolumns: Array = Array(col1, eventdate) Type in expressions to have them evaluated. Using Scala version 2.12.10 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_2xx Spark context Web UI available at Spark context available as 'sc' (master = yarn, app id = application_1234567890123_111111). Is there a more direct approach or am I running into a bug here? My objective is to SHOW PARTITIONS of my EXTERNAL managed Delta table created by another program that is not managed to begin with. The SHOW CREATE TABLE content did not match the original DDL statements I provided and it is causing the SHOW PARTITIONS to fail. I have provided extra debugging information via Spark SQL: DESCRIBE TABLE EXTENDED and SHOW CREATE TABLEĪ.I execute Spark SQL: SHOW PARTITIONS and it failed saying the table is not partitioned.I have successfully created an EXTERNAL table reference as a managed table.This is done explicitly using the logic outlined here
I have created (managed) table reference to the above Delta Table.Ī.
#GLIMS ON THE STUFF FULL#
Using spark-shell from precompiled OSS Apache Spark 3.0.2 without Hadoop + io.delta:delta-core_2.12:0.8.0.
0 kommentar(er)
