Column does not exist in BatchData when column name has a dot (.) - Spark only

[larruda]

Describe the bug
I'm using a SparkFileSystemDatasource with a CSVAsset (header=True). The csv file has a header line and some column names have dots (e.g. Data.Entrega). GX is not able to identify those columns and throw "The column "Data.Entrega" in BatchData does not exist"

To Reproduce
It's an ephemeral context, also I'm not allowed to share any code due to NDA with customer. It's simple as I described, reading a csv asset from a spark datasource with header=true and dots in the col names, simple as that anyone should be able to easily reproduce.

Expected behavior
The column should be found.

Environment (please complete the following information):

• Operating System: MacOS
• Great Expectations Version: 1.4.5
• Data Source: Spark CSV file
• Cloud environment: N/A

Additional context
Add any other context about the problem here.

[larruda]

If I remove the dot from the col name in the csv and also in the expectation it works - which proves the bug.

[larruda]

just found out if I surround the columnb name with backticks ( `Data.Entrega` ) it works fine.
Maybe it has something to do with how spark deals struct columns.
Will leave this opened so you can decide whether it's a GX issue or dot. At least it deserves a note somewhere.

[adeola-ak]

Hi! This occurs because Spark column names that contain special characters are not currently parsed correctly. Since there is a known workaround, this isn’t something we’re able to prioritize right now.

That said, you’re welcome to open a PR with a proposed fix and we’d be happy to review it. In the meantime, continuing to use the workaround you discovered is the best path forward.

We apologize for the inconvenience and appreciate you bringing this to our attention.