The API Jobs Service has fully recovered. All systems are back to normal. Thank you for your patience during this incident.

We are currently experiencing service degradation affecting API jobs. This issue is caused by an incident with one of our subprocessors. Our on-premise infrastructure is unaffected. We are actively investigating and will provide updates as more information becomes available.

We sincerely apologize for the disruption caused by two service outages over the weekend.

Incident 1

Time: Saturday, June 6, from 07:07 to 07:50 UTC

Impact: Our API and Dashboard were unavailable. Job submission and job processing were affected across the platform.

What happened: Core services stopped responding correctly due to elevated database load caused by long-running job-listing queries.

Mitigation: We restarted affected services and terminated the long-running database queries to reduce load and restore service. We then continued monitoring while investigating the underlying cause.

Incident 2

Time: Sunday, June 7, from 00:13 to 03:40 UTC

Impact: A similar issue occurred. Our API and Dashboard were unavailable, and job submission and job processing were affected across the platform.

What happened: Database load increased again from a similar query path, causing core services to stop responding normally.

Mitigation and resolution: We terminated the long-running queries again and implemented an additional fix to reduce load from this query path. Services recovered, and we have not observed the issue reoccur since.

Follow-up actions

We are applying additional improvements to reduce the likelihood and impact of similar incidents:

• Improve database query performance
• Add stronger monitoring and alerting for long-running queries and elevated database load
• Improve incident response procedures to reduce time to mitigation
• Review platform resilience so core services remain more stable under database pressure

Current status

The platform is operating normally. No action is required from users.

We sincerely apologize again for the inconvenience this caused.

Job processing has fully recovered. This incident is now resolved.

We are seeing recovery in job processing and are continuing to monitor the situation.

We are currently experiencing degraded performance on job processing due to an issue with a subprocessor. Users may encounter elevated error rates (500/503s) when attempting to access the API. We are actively investigating the issue and will provide updates as soon as possible.

The issue has been resolved.

We are experiencing some slower than usual job processing due to degraded performance of a subprocessor.

The issue is now resolved

The issue has been identified and is being resolved

We are experiencing some issues with our API and Dashboard

The issue has been resolved. API and job execution service is back to normal.

API and job execution are now recovered and service errors are going down. We're monitoring the situation.

We have issues with our downstream provider.

The issue is resolved. We'll continue to monitor the situation.

We've resolved the issue and are monitoring. Job submissions now go through again.

We're experiencing degraded performance for new job submissions. Jobs might take longer to complete. We are investigating this issue.

We are doing maintenance causing 1 hour of downtime from 12:00AM - 01:00AM UTC January 15