Near’s Data API, whether access through a UI or programmatically is a powerful tool that allows the export of human movement data. Like all software, there are some best practices to follow to ensure that you can derive the maximum possible value in the fewest steps. This article outlines some of the accumulated best practices for using the Data API. As always, contact our support team with any questions.
Common Job Delay Reasons
There are many reasons that a Data Export job can seem delayed. Here are the most common, with suggestions for mitigation:
- Length of Analysis: Jobs can be delayed or appear to be delayed if the analysis time period is extended. Breaking jobs up by time is a way to get data in hand faster and further analysis to begin.
- Company Queue: When a submitted job hits a complexity threshold any subsequent job enters a queue behind the complex job. The complexity threshold is driven by both the count of locations (or total size) of the study area and the type of job requested. Jobs which take significant computing time—pathing, OGS, Location affinity, retail heartbeat are subject to queuing. To avoid queuing, consider splitting out job requests to run less complex reports in one job and the more complex reports in a second job, allowing you to begin using the data while the more complex job completes.
- Complex Shape Definition: The number of points used to define a location shape will generally slow down the processing of determining which points are within a polygon. Mathematically, simpler shapes make for a quicker visitor determination and quicker job finish.
- General API busyness: At times, a lot of companies have submitted jobs to the API. While we have a per company queue, at times, the API can get backed up. Near generally works to increase capacity during these times as quickly as possible, but this rare case can sometimes cause delays.
Common Job Failure Reasons
Geography Size
When Data API jobs are submitted which cover a large geographical area, there is a significantly increased chance that the job will eventually run into a failure point. This is due to the fact that the amount of raw data being processed grows too large and outgrows the storage capability of the servers which run the Data API software. Note that in this case, the phrase “large geographical area” can either be one large polygon or shape (The entire New York City area) or 1000s of individual polygons—such as all the McDonalds in the United States.
In general, to ensure success for your job, follow the following guidelines:
- For larger analysis such as cities or high streets, break up jobs into smaller areas and then stitch the resulting files back together.
- For more than ~1000 places, split your job into multiple jobs. The credit price is determined by number of reports and places, not the number of jobs.
- ESRI conversion failures: if you have selected a conversion to geoDB formatted output, large jobs can sometimes fail due to errors in the underlying open-source software used to do the conversion. Again, splitting the jobs becomes more critical when using ESRI.
Unsupported Reports per Country
Not all reports are available in all countries. Jobs will fail if a report for a even just one location in unsupported country is requested as part of the job. As of June 2022, the follow country availability is in place:
- Estimated Visits: supported in US, CA, AUS, NZ
- Extrapolated Visits: US
- Demographics: US and AUS. Additional countries may be available by license.
- Retail Heartbeat: US and CA
- Geosocial Affinity: US
- Location Affinity: US and CA
Date Restrictions
Some reports require additional processing (for example of panels and baselines), and have restricted available date ranges: • Retail Heartbeat is available from January 1, 2018 until the day before the beginning of the previous month. • Estimated Visits 2.0 is available from January 1, 2019 until 5 days before today.
Any report requested outside of these timelines will generate a job failure.