Unique / Array Layer
Get unique values from fixed-length arrays while maintaining order. Similar to Python's dict.fromkeys() or R's unique(). Creates a new array with duplicate values removed.
Example transformation (arrays of length 4):
| arrays | unique |
|---|---|
| [1, 2, 2, 1] | [1, 2] |
| [a, b, a, b] | [a, b] |
| [2, 2, 2, 2] | [2] |
| [1, null, 1, null] | [1, null] |
Common applications:
- Removing duplicates
- Finding distinct values
- Data deduplication
- Set operations
- Unique identifier extraction
Note: Arrays must have the same fixed length across all rows. Output array length will vary based on number of unique values. Preserves order of first occurrence.
Table
0
0
Table
Select
columnThe fixed-length array column to deduplicate. Supports arrays of any type:
- Numeric arrays: [1, 2, 2, 1] → [1, 2]
- String arrays: [a, b, a, b] → [a, b]
- Boolean arrays: [true, true, false, false] → [true, false]
- Date arrays: [2024-01-01, 2024-01-01, 2024-01-02, 2024-01-02] → [2024-01-01, 2024-01-02] All input arrays must have the same length
AsColumn
nameName for the new column. If not provided, the system generates a unique name. If AsColumn matches an existing column, the existing column is replaced. The name should follow valid column naming conventions.