How an ‘Army of Robots’ Is Transforming Open Data in NYC

New York City is using automated data feeds and streamlined processes to improve transparency. The innovative approach offers lessons for other governments seeking to avoid costly legal battles and promote public trust.

New York City is a global leader in open data, with a massive portal containing more than 2,500 data sets and 6 billion rows of information, ranging from city employee test scores to restaurant inspection results. But building and maintaining the collection has been a journey of evolution and ongoing improvement, one that’s increasingly becoming automated.

In New York City, open data is part of legislation passed by former Mayor Mike Bloomberg in 2012, mandating that all public data be published online on a single portal by 2018. To improve the portal, the Open Data Law was amended in 2015 and 2016 to require regular updates to data sets and careful review of Freedom of Information Law (FOIL) requests for potential new data sets.

The ambitious legislation has led to a robust portal, but it’s not perfect. According to the city’s dashboard, only about 60 percent of data sets are updated on time. Additionally, only about 40 percent of planned data set releases have appeared on the portal on time in the last year.

While open data in the city continues to evolve, the moves it’s making to ensure data accuracy and accessibility offer lessons for other governments. Many municipalities struggle to comply with public information laws due to overwhelming requests, often leading to costly legal battles. For example, reporting from the San Jose Spotlight revealed that San Jose, Calif., has been sued at least six times for failing to provide public information, costing the city more than $500,000 in taxpayer dollars after losing several cases.

BetaNYC, a nonprofit civic organization that was involved with lobbying for New York City’s open data law, asserts that open data portals shouldn’t be considered optional.

“If your city doesn’t have an open data portal, they’re wasting your tax dollars and time,” said Noel Hidalgo, executive director of BetaNYC. Hidalgo added that the costs the government incurs by putting every data request through a lawyer or communications representative can often be avoided by making appropriate data available to the public at all times. “There are more efficient ways to share information in the 21st century.”