Is there a way to create a minute-level partition in BigQuery? I ask because we are getting a high volume of iot data every minute from our nprobe server. We need to process the data at minute level.

Is this correct?

CREATE OR REPLACE TABLE your_dataset.your_table
PARTITION BY TIMESTAMP_TRUNC(event_timestamp, MINUTE) AS
SELECT
    event_id,
    event_timestamp,
    event_data
FROM
    your_source_table;

Is there a way to create a minute-level partition in BigQuery? I ask because we are getting a high volume of iot data every minute from our nprobe server. We need to process the data at minute level.

Is this correct?

CREATE OR REPLACE TABLE your_dataset.your_table
PARTITION BY TIMESTAMP_TRUNC(event_timestamp, MINUTE) AS
SELECT
    event_id,
    event_timestamp,
    event_data
FROM
    your_source_table;

• google-bigquery

1 Answer 1

BigQuery doesn't support minute partition. Also if you really partition by minutes, per the maximum 10K partitions per table limit, you can only keep 7 days of data in the table.