Everything is now back to normal, apologies again for the interruption there!
Jan 10, 15:30-15:56 UTC
Some Apps Experiencing Issues
InvalidParameterValueException errors have been resolved - private and invite only developer apps are back to running like normal.
Jan 6, 01:51-03:09 UTC
Everything seems to be back to normal.
Dec 22, 15:59-17:59 UTC
Partial 15 Minute Outage & Task Delays/Loss
Between Sunday December 18th 21:54:00 and 22:09:00 UTC (a 15 minute window approximately 6 hours before this post) we experienced a network partition in our RabbitMQ cluster resulting in some lost or delayed tasks. The details are as follows:
Any non-instant (polling) Zaps with tasks triggered in that 15 minute window likely experienced up to a 6 hour delay as we recovered any pending tasks from logs. No tasks were lost in this case. Most Zaps fall under this category.
Any instant Zaps with tasks triggered in that 15 minute window either experienced a maximum of 60 minutes delay and possibly a double task OR were lost. If your webhooks were delivered to the newer, safer hooks.zapier.com endpoint they were simply delayed - however, if your hooks were delivered to the legacy zapier.com endpoint - they were lost. Any instant Zaps turned on within the last 3-6 months are under the newer, safer hooks.zapier.com endpoint. Any instant Zaps turned on before 6 months ago are likely on the legacy zapier.com endpoint.
We've been working to move all instant Zaps over to the newer, safer hooks.zapier.com endpoint -- you can force your Zaps over by turning off and on your Zap and (if applicable) updating any webhook URLs pointing to Zapier in your connected apps.
In the last 24h approximately 1% of all tasks were delayed a significant amount of time (more than 15 minutes), and significantly less than 1% of webhook tasks were lost. To ensure this doesn't happen again we will be taking the following actions as a direct response:
1. Correct suboptimal configurations and assumptions in our RabbitMQ failover strategy.
2. Hasten the already in-progress migration from legacy zapier.com webhook endpoints to the newer, safer hooks.zapier.com endpoint.
3. Correct a bug that improperly signaled a failure to partners when a webhook was properly journaled to disk but not to RabbitMQ (this risked double tasks in a few cases).
We apologize for the interruption, we've been very proud of our reliability and transparency improvements throughout 2016. We're going to do our best to ensure this never happens again.
Dec 19, 04:12 UTC
Minor Site Outage
Site is now fully operational. No Zaps or Tasks were effected.
Dec 9, 00:06-00:11 UTC
ActiveCampaign has informed us that the underlying problem has now been resolved.
Nov 25, 18:30 - Nov 28, 18:14 UTC
Delayed Write Tasks
Queue backlog is now caught up and tasks that were delayed should execute by now.
Nov 21, 19:33-19:53 UTC
Daylight Savings Bug
Everything thing is looking great again - just a few more tasks to catch up on. Email firstname.lastname@example.org if you notice any tasks stuck for longer than 2 hours.
Nov 6, 06:34-07:17 UTC
All writes and tasks are back up to speed - no tasks were lost!
Nov 1, 18:27-19:00 UTC
Oct 24, 21:16-21:31 UTC
Zaps & Tasks Slowing Down
Everything is back to normal. During the blip we saw very few errors but many delayed/slow tasks - our apologies! All tasks should be running like normal again.
Oct 24, 20:15-20:56 UTC
All zaps and tasks are back up to speed. As long as DynDNS holds steady we should be okay! We're preparing an email to go out to those without auto-replay enabled that experienced errors with remediation instructions. Expect those within the next hour or so.
Oct 21, 16:23-17:44 UTC
And we're back to normal! Thanks for bearing with us, and sorry for the disruption today.
Oct 21, 11:54-15:01 UTC
Delayed Instant Triggers
Instant triggers are now back to full speed. Sorry for the interruption!
Oct 18, 13:13-13:46 UTC
Zapier.com is now back to normal - sorry for the interruption!
Sep 27, 12:54-13:11 UTC
Triggering on old Evernote notes
Between 2016-09-20 20:19:00 and 2016-09-20 20:36:00 UTC, a bad deploy caused a small percentage of Evernote Zaps to incorrectly trigger on older notes. The issue has been resolved and Zaps are back to their normal polling schedules.
Sep 20, 21:56 UTC
Task Delay Issues
A bad deploy between 2016-09-14 20:00:00 and 2016-09-14 21:45:00 UTC broke some vital behavior, causing a big backup in our queue and a small number of tasks to be lost (less than < 1 % of instant zaps).
If you visit https://zapier.com/app/history?status=pending and see any tasks you should be replay them. Email email@example.com if you have any problems there or suspect we may be missing other tasks during the same time period.
Sep 14, 21:35-22:39 UTC
Slow API response times
Slow response times with the Google Sheets API began to cause back-ups.
Sep 12, 21:21-22:06 UTC
BurstSMS migration to AWS was successful. From BurstSMS: "The migration was successful. As a precaution, we kindly ask all customers to please re-test all your workflows and system connectivity."
Sep 9, 22:00 - Sep 10, 02:37 UTC
Gmail's APIs have slowed down tremendously again. Gmail related tasks will be delayed.
Aug 31, 23:07-23:47 UTC
Google Sheets API Slowdown
We're seeing a significant slowdown in google sheets calls. As such Google Sheets dependent tasks may be delayed.
Aug 31, 19:53-21:34 UTC
Active Campaign webhooks are now being delivered, however we are still waiting to hear if missed hooks will be redelivered.
Aug 23, 15:00 - Aug 24, 02:46 UTC
Everything is back to normal!
Aug 23, 16:20-22:00 UTC
This incident has been resolved.
Aug 23, 15:45-16:05 UTC
Issues with Gmail Zaps
The issue with Gmail Zaps is now resolved, and things are back to normal. Zaps will pick up any emails that were delayed during the interruption. Sorry for the hiccup!
Aug 17, 14:13-14:50 UTC
Code by Zapier Impacted
AWS issues are now resolved, Code by Zapier is now operating normally again.
Aug 11, 02:09-03:20 UTC
Tasks writes have caught up and running at their normal speed.
Aug 4, 20:41-20:45 UTC
Eventbrite Hooks are Down
Eventbrite has resumed sending webhooks to zaps and should be functioning as intended moving forward.
Jul 29, 17:37-23:42 UTC
The issues with our provider are now resolved, and Zaps are running at full speed again. Any failed tasks may be replayed; customers with Autoreplay enabled need take no action.
Jul 26, 10:03-10:21 UTC
Feedly Zaps Broken
Feedly Zaps are back to normal, we apologize for any Feedly Zaps that triggered unexpectedly and we are prioritizing internal safety mechanisms to prevent this from happening again. Below is recap of what happened and what we're doing now.
The anomaly was caused by a broken migration that changed how Zapier "identifies" new posts in Feedly triggers. This caused Zapier to think every post it saw was "new" and therefore required triggering. Once we identified this issue, we immediately began halting Feedly Zaps and initiating a rollback. After the rollback, we confirmed Feedly Zaps were behaving properly again and began resuming Feedly Zaps with the correct behavior. All Feedly Zaps are back to normal.
Any "Pending Tasks" for Feedly you may see in your Task History will not run.
As a result, we'll be prioritizing the implementation of new safety mechanisms to validate the behavior of any migrations so that this doesn't happen again. We apologize for the hassle.
Non-Feedly Zaps were not effected in this outage.
Jul 22, 20:50-21:19 UTC
Upstream Provider Network Issue
Networking issues look fully resolved - we're all clear now and zaps are back up to speed!
Jul 19, 18:57 - Jul 20, 03:20 UTC
Increased Gateway Timeout Errors
We're seeing an uptick in gateway timeouts on the survey monkey side. We'll be monitoring the situation and allow SurveyMonkey specific tasks to resume once all is clear.
Jul 12, 19:32 - Jul 13, 08:56 UTC
SSL Validation Errors
SSL handshake errors have ceased and Freshbook tasks are now flowing once again.
Jul 11, 18:36-18:51 UTC
Delayed Instant Triggers
The instant triggers backlog has been processed, and we're back up to full speed!
Jul 8, 20:04-20:22 UTC
Task Processing Delays
The upstream network issue effecting our task processing speed has been resolved; we'll be back up to full speed in 10 minutes!
Jul 7, 16:09-16:24 UTC
Evernote API Slowdown
Evernote polls have slowed down tremendously. Keeping an eye on it, some polls may be delayed.
Jul 2, 22:13-22:33 UTC
Google Calendar issues
Errors are now low enough that this incident is resolved. Users with paid plans can replay tasks that failed as a result of the Google Calendar outage, and those with Autoreplay enabled needn't take any action.
Jun 30, 14:07-16:31 UTC
Facebook Lead Ads Errors
We're unsure of the root cause of this outage from Facebook Lead Ads - we're awaiting a response from them but it appears the errors stopped 2016-06-29 10:37:35 UTC.
Jun 29, 05:09-15:58 UTC
[Scheduled] Database Maintenance
That's a wrap! We've finished up our database maintenance and polling is gradually coming back up to speed. We'll be at full speed in about 20 minutes.
Jun 23, 04:01-05:29 UTC
Currently experiencing some gateway timeouts on Typeform's side.
Jun 17, 17:44-18:23 UTC
Typeform Gateway Timeouts
Typeform is currently having a lot of gateway timeouts. Polling may be delayed.
Jun 13, 17:49 - Jun 14, 08:41 UTC
Gmail trigger issues
Gmail triggers are back to normal. Sorry for the interruption!
Jun 13, 14:07-14:57 UTC
Code by Zapier issues
The underlying service interruption has now been resolved, and all Zaps are running normally again. Any failed steps can be replayed, and will be automatically for customers with Autoreplay enabled.
Jun 9, 19:17-21:32 UTC
Back End Networking Issues
Everything back to 100% on backup systems. Maintenance starting shortly - stay tuned!
Apr 21, 21:11 - Apr 22, 03:46 UTC
Isolated Google Sheets, Calendar & Mail Issue
Google Mail, Calendar, and Sheets error rates have subsided and are working properly again. If you don't have autoreplay enabled, consult your task history to replay any failed tasks.
Apr 2, 18:41-19:14 UTC
Isolated Active Campaign Issue
This appears to have resolved. We're continuing to monitor the situation but Active Campaign Zaps should be back up and running!
Apr 1, 23:40 - Apr 2, 18:42 UTC
JotForm is operating normally again. If you don't have Autoreplay enabled, consult your task history to replay any failed tasks.
Mar 31, 18:36-19:24 UTC
Code by Zapier issues
AWS issues are now resolved, everything seems to be operating normally again.
Mar 31, 03:42-05:21 UTC
Zapier.com and Task Slowdown
The underlying slowdown issue has subsided, and we're back up to full speed running Zaps and Tasks.
Mar 30, 20:14-20:19 UTC
A fix has been deployed and tasks can be replayed.
Mar 30, 00:49-04:23 UTC
Everything seems to be back to normal.
Mar 25, 08:36-09:08 UTC
SMS by Zapier Outage
The "SMS by Zapier" app experienced an outage between Wed, 23 Mar 2016 2:40:00pm PST and Wed, 23 Mar 2016 5:45:00pm PST that has since been resolved. We will send an email to affected users soon with any information on recovery steps. No other apps or services were affected.
Mar 24, 00:46 UTC
Todoist is operating normally again. If you don't have autoreplay enabled, consult your task history to replay any failed tasks.
Mar 23, 13:21-13:30 UTC
Delayed instant triggers
We've cleared the backlog and instant triggers are operating normally again.
Mar 22, 13:42-14:06 UTC
Facebook Leads Issue
Over the past few days Facebook Leads have had intermittent delivery issues. We've been in contact with Facebook and they assure us that the underlying issue has been resolved as of Mon, 21 Mar 2016 5:30:00 PM PST. If you are still missing leads after that time - please contact us https://zapier.com/app/contact-us.
Mar 22, 00:46 UTC
Isolated Trello 404 Errors
A Trello DDoS protection load balancer was blocking some Zapier worker IPs, returning 302 redirects which would 404 on redirect. This issue has been resolved by Trello.
Mar 10, 00:19 UTC
Sporadic Errors for Some Zaps
Over the last 36 hours while upgrading our monitoring systems a bug caused a small number of tasks to hit errors. While no tasks were lost, tasks may be stuck in an error state --- recovery emails with replay instructions have been sent to to all effected users --- if you didn't get an email, you weren't effected.
Mar 9, 22:27 UTC
Instant Trigger Outage
Scheduled maintenance hit a snag. Polling Trigger Zaps were delayed by up to 50 minutes (but all completed successfully) and Instant Triggers were lost during a window of approximately 25 minutes. Emails have been dispatched to those effected - check your inbox for more information.
Mar 1, 19:15 UTC