Tracking rates of change of Open Data Portals¶
Many of our reports note that most open data is ephemeral and that it regularly disappears or is overwritten. We informally tracked this rate of change during two periods.
Warning
These are internal numbers that are for BTAA-GIN program tracking purposes. They should not be used for research or analysis in other contexts. Contact the data providers to verify this information for other projects.
March 2017 to December 2018¶
From March 2017 to December 2018, the BTAA GDP Metadata Coordinator tracked the number of items that were deleted and added to 15 ArcGIS Open Data Portals (now calle "Hubs). These were ingested on a monthly basis into the BTAA Geoportal.
Note
Cells with a dash "-" indicate that the portal had not been added to our collection yet.
Items Deleted by selected ArcGIS Open Data Portals¶
| Date | 2017-03 | 2017-04 | 2017-07 | 2017-08 | 2017-09 | 2017-10 | 2017-11 | 2017-12 | 2017-01 | 2018-02 | 2018-03 | 2018-04 | 2018-05 | 2018-06 | 2018-07 | 2018-11 | 2018-12 | Sum |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| District of Columbia Open Data | - | - | - | 2 | 0 | 0 | 49 | 225 | 17 | 2 | 5 | 5 | 3 | 0 | 26 | 137 | 33 | 504 |
| GeOhio | - | - | - | 0 | 0 | 0 | 0 | 0 | 20 | 16 | 7 | 78 | 12 | 4 | 0 | 174 | 20 | 331 |
| Harford County GIS Data | - | - | - | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 33 | 33 |
| Indianapolis Open Data | - | - | - | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 6 | 1 | 2 | 0 | 0 | 0 | 2 | 11 |
| IowaDOT | - | - | - | 10 | 0 | 19 | 3 | 1 | 4 | 0 | 1 | 8 | 0 | 0 | 0 | 203 | 16 | 265 |
| Lake County Illinois Open Data Portal | - | - | - | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 362 | 156 | 518 |
| Maryland iMap | - | - | - | 11 | 3 | 2 | 1 | 1 | 1 | 1 | 9 | 1 | 4 | 0 | 3 | 5 | 3 | 45 |
| Michigan DNR | - | - | - | - | - | 6 | 11 | 13 | 15 | 77 | 2 | 1 | 2 | 0 | 73 | stopped harvesting | - | 200 |
| Minneapolis Open Data | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | 16 | 16 |
| Peoria County GIS Open Data Portal | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | 1 | 1 |
| SEMCOG | 0 | 8 | 0 | 0 | 0 | 4 | 0 | 0 | 0 | 9 | 0 | 0 | 0 | 0 | 1 | 0 | 0 | 22 |
| State of Michigan Open Data Portal | 62 | 0 | 133 | 165 | 0 | 6 | 14 | 13 | 18 | 80 | 2 | 4 | 2 | 0 | 73 | 5 | 8 | 585 |
| Washington County, MN | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | 2 | 0 | 2 |
| Sum | 62 | 8 | 133 | 188 | 3 | 37 | 78 | 253 | 75 | 186 | 32 | 98 | 25 | 4 | 176 | 888 | 292 | 2538 |
Items Added by selected ArcGIS Open Data Portals¶
| Date | 2017-03 | 2017-04 | 2017-07 | 2017-08 | 2017-09 | 2017-10 | 2017-11 | 2017-12 | 2017-01 | 2018-02 | 2018-03 | 2018-04 | 2018-05 | 2018-06 | 2018-07 | 2018-11 | 2018-12 | Sum |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| District of Columbia Open Data | - | - | - | 9 | 0 | 0 | 62 | 3 | 25 | 10 | 16 | 22 | 32 | - | 27 | 137 | 33 | 376 |
| GeOhio | - | - | - | - | - | - | - | - | 54 | 14 | 4 | 29 | 17 | 29 | 0 | 173 | 20 | 340 |
| Harford County GIS Data | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | 35 | 35 |
| Indianapolis Open Data | - | - | - | - | - | - | - | - | - | - | 1 | 0 | 5 | 0 | 0 | 0 | 3 | 9 |
| IowaDOT | - | - | - | 3 | 0 | 17 | 16 | 3 | 19 | 0 | 0 | 3 | 0 | 0 | 0 | 138 | 27 | 226 |
| Lake County Illinois Open Data Portal | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | 362 | 140 | 502 |
| Maryland iMap | - | - | - | 17 | 0 | 7 | 0 | 9 | 22 | 168 | 11 | 6 | 5 | 0 | 12 | 6 | 7 | 270 |
| Michigan DNR | - | - | - | - | - | 20 | 6 | 105 | 102 | 5 | 9 | 5 | 26 | 0 | 31 | retired from portal | - | 309 |
| Minneapolis Open Data | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | 0 | 0 |
| Peoria County GIS Open Data Portal | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | 0 | 0 |
| SEMCOG | - | 15 | 0 | 0 | 0 | 3 | 0 | 0 | 0 | 5 | 0 | 0 | 0 | 0 | 1 | 0 | 0 | 24 |
| State of Michigan Open Data Portal | 20 | 0 | 0 | 254 | 0 | 27 | 14 | 105 | 110 | 10 | 9 | 8 | 26 | - | 31 | 9 | 3 | 626 |
| Washington County, MN | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | 5 | 0 | 5 |
| Sum | 20 | 15 | 0 | 283 | 0 | 74 | 98 | 225 | 332 | 214 | 50 | 73 | 111 | 29 | 102 | 830 | 268 | 2724 |
March - September 2023¶
By 2023, our metadata collections had grown considerably and we again tracked the rate of change for:
- 67 ArcGIS Hubs
- 1 custom state data portal (PASDA)
- 3 CKAN data portals (State of Minnesota, OpenDataPhilly, and CMAP (Chicago Metropolitan Agency for Planning) Data Hub
2599 total datasets were removed from the portals during that time and, presumably, replaced.
| Month | Number deleted |
|---|---|
| March 2023 | 94 |
| April 2023 | 148 |
| May 2023 | 92 |
| June 2023 | 213 |
| July 2023 | 150 |
| August 2023 | 780 |
| September 2023 | 1122 |