I’m working with a large dataset that I fetch using this code:

    response = await client.get(tenant_url, headers=headers, params=params)
    response.raise_for_status()
    data = response.json()
    numpy_2d_arrays = np.array([[device["id"]["id"], device["name"]] for device in data["data"]])

To improve performance, I’m considering converting the data to a Polar DataFrame before extracting the id and name. Will this help improve performance, or is there a better way to handle large datasets in loops?

I’d appreciate any suggestions or performance tips!

data is

{
  "data": [
    {
      "id": {
        "entityType": "DEVICE",
        "id": "7c4145c0-e533-11ef-9681-df1aaf416822"
      },
      "name": "C4DEE264E540-002"
    },
    {
      "id": {
        "entityType": "DEVICE",
        "id": "be7b4b90-b36a-11ed-9188-71dcc7f44f0f"
      },
      "name": "C4DEE264E540"
    },
    {
      "id": {
        "entityType": "DEVICE",
        "id": "fbcded60-de0e-11ef-9681-df1aaf416822"
      },
      "name": "C4DEE264E540-001"
    }
  ],
  "totalPages": 1,
  "totalElements": 3,
  "hasNext": false
}