This can happen in the scenario where table definition and specific partition definition is different, and the underlying data matches table definition but not partition definition.
When a table with partitions is altered to add a column using statement:
ALTER TABLE <tableName> ADD COLUMNS (c1 int);
The table definition for existing partitions don't get modified as per the above statement. As a result of this there is a mismatch between partition and table definition.
This is ok if the partition data matches the definition of partition, but if the data matches definition of table itself, NPE is thrown as there is a mismatch in data vs definition.
To avoid this issue, this statement should be used in hadoop2
ALTER TABLE <tableName> ADD COLUMNS (c1 int) CASCADE;
In case of hadoop1, CASCADE option is not available. Hence, as long as the table is external table, following can be done:
- Drop and recreate partitions for this table
- Alter partition definition for specific partition having issues