You should upgrade to 3.6.3 as it should be better and solve your issue. Note that it will need the web app to be 3.6.3 as well.

Alright, after hours of researching and debugging, I figured out the issue. The scheduler service is just not very efficient at batching emails when there are thousands of users, dozens of projects, and dozens of watchers on a variety of projects.

I did some HTTP capturing of the Gemini Scheduler services (both 3.5 and 3.6) and there are a LOT of calls made.

For IWA setups, EVERY call requires 2 calls to the web service.

1.) Original GET/DELETE http call, which results in a 401
2.) Additional call where IWA kicks in.

So, multiply everything by 2 to start.

The service pulls a list of all pending alerts.

• For Each Alert:
• Pull Issue Data
• Pull Issue Alert Data (get list of people who should be alerted)
  
• Pull Issue Data Again (why??)
• For Each Person Who Should Be Alerted:
• Pull User Information
• Pull Issue Data Again (why??)
• Pull Issue Comment Data if necessary

Anyway, it turns out that it takes something like 2 minutes and 10MB of data to send out the alerts on ONE ticket on a project that has many watchers. We get 5-10 tickets a minute during the busy time of our day.

Upgrading our production system to 3.6 at this time is not an option, but after installing a test version of 3.6 and migrating the data and analyzing the traffic, I don't really see any improvement. I don't understand why the service needs to pull the same issue's data over and over and over again for one alert.

I don't see any good options to resolve this:
1.) Disable users from watching a project (bad idea).
2.) Disable Scheduler and use MailPlugin (spammy)
3.) Pay someone to create a more efficient scheduler (expensive)
4.) Bug reports, wait for new release, then upgrade our system
5.) ???

Any ideas?

This is becoming very frustrating.

I removed ALL project watchers from all projects. I ensured that "Email Issue Creator" and "Email Issue Resource" are enabled in Administration.

But now Scheduler will not send out any emails, not even to the creator or resource, if they are not watching the project.

So basically, for a Creator to get an email notifying him that the ticket was created:
1.) Email Issue Creator must be on
AND
2.) Creator must have "Receive alert when issue is CREATED" selected on the project.

But, selecting #2 causes an email to go out for ALL CREATED issues on that project, which is NOT what I want. I want users to ONLY get an email for issues they created without having to monitor the entire project.

It turns out, if I have all notification disabled on a project, and I watch an issue manually, I still will not get any alerts for that issue because I don't have any checked on the project. This is very counter-intuitive - basically my Option #1 above does not work; I am forced to have user select project level notifications in order to get issue level notifications, which caused this problem to begin with.

I guess I'll just have to stick with the spammy MailPlugin for now and the users will have to deal with getting notifications for issues they aren't related to until #3 or #4 happen.

...and it turns out it was a configuration issue. We had WATCHER alerts disabled, so all users were subscribing to project level alerts to get their notifications and Scheduler was having to do a lot of work to figure out who to email. We mistakenly believed that WATCHER alerts would send a notification when a watcher was added or removed, not that it enabled alerts for watchers. The Gemini documentation is not clear about this.

To resolve, I deleted all existing alert subscriptions, enabled watcher alerts, and updated the text at the top of the "Configure Project Email Alerts" to say:

Choose the project level notifications you wish to receive.
Important: These settings apply to ALL issues in the project!
For example, if you enable "Receive alert when issue is CREATED", you will receive an email for every issue created in this project.

Thanks for the update, glad all is well now.

We will revise our docs.

Okay, now that we're back to high volume I'm still seeing issues. It's getting pretty slow in processing notifications, and I'm noticing that there always seem to be 2 scheduler processes running.  Is this normal?


At some points, we keep getting the same email over and over - when I look at the server I see 3 or 4 processes running. No errors in the error logs or event logs.

What do you recommend for a scheduler interval setting? The notifications are being added to the alerts table at a rate of 10-20 a minute.

If you have more than 2 instances running (2 is ok) then it means that the interval is too low (too fast).

Set it to 2 as a minimum but 5 minutes should do the trick. If this doesn't help let us know.