What Happened?
Between November 25 and December 2, 2025, our Auto-Broadcast system stopped updating job board feeds and integrations. This meant that job postings were not being refreshed or sent to external job boards as expected for a period of six days.
How Were Customers Impacted?
Customers relying on timely updates to their job boards experienced delays in having their job postings refreshed or distributed. This could have affected the visibility of open positions and potentially delayed candidate applications.
What caused the issue?
- The automated process responsible for updating job board feeds stopped working because a tracking file from a previous run was not deleted as it should have been.
- This leftover file caused the system to believe that updates were still in progress, so no new updates were started.
- The tracking file was not deleted because the system component running the process was unexpectedly restarted before it could finish its work.
- We did not have monitoring in place to alert us when this process failed.
How was the issue resolved?
- Once notified, our support and engineering teams quickly identified the cause.
- The undeleted tracking file was manually removed, allowing the system to resume normal operation.
- It took several hours for the system to catch up on the backlog of updates, but by the next day, all job board feeds were up to date.
What are we doing to prevent this in the future?
- We have implemented automated monitoring and alerts to notify our team immediately if the Auto-Broadcast process stops working.
- We have documented troubleshooting steps to ensure a faster response if a similar issue occurs.
- We are reviewing and improving the system to prevent similar failures from happening again.