InternalError_: Spectrum Scan Error. S3 to Redshift copy command

Question

I am trying to copy some data from S3 bucket to redshift table by using the COPY command. The format of the file is PARQUET. When I run the execute the COPY command query, I get InternalError_: Spectrum Scan Error.

This is the first time I tried copying from a parquet file.

Please help me if there is a solution for this. I am using boto3 in python.

Solution 1

This generally happens for below reasons:

1. If there is a mismatch in number of columns between table and file.
2. If the Column type of your file schema is incompatible with your target table column type.

Try going into the error logs. You might find partial log in cloud watch. From the screen shot you have uplaoded, you can also find a query number you have run.

Got to aws redshift query editor and run below query to get the full log:

select message 
from svl_s3log 
where query = '<<your query number>>'
order by query,segment,slice; 

Hope this helps !

Solution 2

This error usually indicates some problem with compatibility of data in your file and redshift tables. you can get more insights about error in table 'SVL_S3LOG'. In my case it was because file had some invalid utf8 characters.