Issues with cron emails

[JackGoa]

Hi, I've been experiencing issues with cron emails that don't all go out anymore and I don't know exactly how to go about troubleshooting it. I'll try to explain what I've found.

First of all (not related to this issue, but an issue nevertheless), the time of the crons seem to be on universal time, they don't use the server's time. It's a small thing, but I have noticed this on all sites where I use cron emails.

So, on one site, I have 12 crons setup. 10 are email crons and 2 are php crons. From what I can tell, since the J3.6/Fabrik3.5 update, my email crons will only get about halfway and then the rest don't run. As far as configuration or anything goes, nothing has changed from my side. they were literally working one day, then not the next day.

It seems that sometimes only 4 email crons will run and only 1 php cron, sometimes 6 or only 5, there does not seem to be any consistency like before.

The only entries I can find in the logs are:

Fabrik DB log:

- 8192,mysql_connect(): The mysql extension is deprecated and will be removed in the future: use mysqli or PDO instead,/var/www/xxx/xxxx/httpdocs/libraries/joomla/database/driver/mysql.php,89

- 8,Undefined index: rowid,/var/www/xxx/xxxx/httpdocs/components/com_fabrik/router.php,231

Server error log (a Joomla error. I think the slash before /media/ should not be there, not sure?):

mod_fcgid: stderr: PHP Warning: is_file(): open_basedir restriction in effect. File(/media/system/js/caption.js) is not within the allowed path(s): (/var/www/xxx/xxxx/:/tmp/) in /var/www/xxx/xxxx/httpdocs/libraries/joomla/filesystem/file.php on line 626

Other than that, I have no idea where else to look to troubleshoot this issue? I've been monitoring it for a few days now, and it's definitely an ongoing problem.

I also found this in Fabrik's logs, but fixed it by changing the SMTP settings in Joomla's config. Even though, I don't understand why it suddenly insists on using SSL now? But obviously unrelated to the cron thing, just thought to mention it in case:

2,stream_socket_enable_crypto(): Peer certificate CN=`my.sslserveraddress.com' did not match expected CN=`localhost',/var/www/xxx/xxxx/httpdocs/libraries/vendor/phpmailer/phpmailer/class.smtp.php,344

 

[JackGoa]

So I received 6 email notifications of 6 out of 12 crons that ran last night. If I look at the Fabrik logs in the db, there are 12 entries though! Interestingly, last night's entries alternate, one is an entry about the records that's been updated, then the net entry is this:

8,Undefined index: rowid,/var/www/xxx/xxx/httpdocs/components/com_fabrik/router.php,231

And so it alternates. So, I'm obviously not getting the email notifications for the entries with that undefined index notification. All logic says to me, whatever that entry means, there is a problem there, as that seems to be the entry made when a cron does not run.

Just guestimating here...?

 

[JackGoa]

FYI, running them manually works perfectly fine... just seems like the automation part has a problem.

 

[cheesegrits]

The router.php undefined index has been fixed in github, although I doubt very much it has anything to do with your cron issues.

I really can't do much to help with this kind of issue in Community support.

Are all the crons running on the same schedule?

-- hugh

 

[JackGoa]

They're set to run 5 minutes apart...
I understand Hugh. Will have to chat end of month again then... :/
I'm positive this has something to do with the recent update though. Was fine before then.
9 crons ran tonight FYI...

bj?rn
- from tapatalk

 

[JackGoa]

can confirm, 9 of my ten cron emails ran tonight and not any of the two php crons.

 

[JackGoa]

This is getting confusing. On one of my other sites, I noticed one of the recent email crons I created was unpublished. So I published, it ran immediately. It sent out the emails, but did not post anything in the log, nor did it update the column it is supposed to be updating, so twice in a row, it sent the same email as it did the day before! And then it unpublished itself again?

Does not make any sense? But something is definitely wrong. This is on more than one site that I am experiencing cron issues now and it's really annoying emailing the same people the same email, 3 times, two days in a row.

 

[JackGoa]

I'm still experiencing cron email problems. Have the same issue on a new site I setup now. Pulling my hair out. Cannot figure out what is wrong. On this new site the emails don't even go out at all, even when I run the cron manually.

*edit - on this new site, I managed to fix the issue by adding, _raw, to one of the fields in my conditions. So remains to be seen how this one goes. The other issues in this thread still exist though.

 

[JackGoa]

Found some errors about read data timeout in 45 seconds and premature end of script headers in the error log files for all the sites running crons. More or less at the time that the script is supposed to run. I changed the max_timeout and max_input_time PHP settings for those sites to 120. Think I might be onto something. Can't see why it would take that long to send 10 emails at a time though, but it is a matching pattern, so lets see.

Any ideas on how to get the scripts to keep running at the times they are set to run? Also, to get them to actually run at the server time and not the global time?

 

[cheesegrits]

Can't see why it would take that long to send 10 emails at a time though

What mechanism are you using to restrict it to 10 rows?

-- hugh

 

[JackGoa]

It's not that I'm restricting it, it's just usually plus minus 10 emails per cron being sent.

I send out reminders, 10 in total, at 5 minute intervals once a day. There's usually around 10, mostly less that go out per cron.

 

[cheesegrits]

Right, but how is that controlled? Is the list only 10 rows, or are you using a condition to decide which ones to send?

Remember that unless you are pre-filtering, or using the "max rows" setting, the cron plugin is going to load the entire table, all rows, and render the elements (as if it was going to be displayed in a list), and hand the email plugin a $data structure with ALL those rows in it. So if your table has (say) 1000 rows, and you are using the condition code to decide which ones need to be sent ... you are loading, rendering and testing 1000 rows, even though you only send email to 10. That's like displaying a List with a 1000 rows in it.

What I typically do, if I have a table with some "email_sent" type flag in it, is rather than use a condition in the cron plugin to figure out if email_sent is set, I'll copy the List, and put a pre-filter on it for "where email_sent != 1", then use that copy for the email cron. That way, it ONLY loads the rows I want.

-- hugh

 

[JackGoa]

I'm not 100% sure about everything you mentioned, but this is my condition code:

$date1 =new DateTime();
$date2 =new DateTime('{ahsa_leads___date_time}');
$timediff = $date2->diff($date1)->format("%a");

return '{ahsa_leads___followed_up_raw}' == 'no' && '{ahsa_leads___form_returned_raw}' == 'no' && '{ahsa_leads___end_result_raw}' == 'pending' && $timediff > 2;

On my other sites it's more or less the same thing, except I use a piece of code you gave me to send an article template as the email instead of a message I entered in the message field.

So in the above code, it checks, has the user been followed up (in the other crons this changes to 'yes', '3' (for three times), etc.
Have they returned their application, if yes.
What is the result of their application.
And then the time difference, which also changes from 2 to 4 to 6 etc in the following crons.

I let the crons run from the bottom up, so the last follow up cron runs first, then the 9th, then the 8th, etc, this I used as a preventative for having them all go to one user in one go, but this does not always seem to work, somehow it seems the crons don't always run in sequence?

 

[JackGoa]

My schedule on this specific site is finished now for the evening. I only received email notifications of 5 crons running. Looking at the Fabrik log in the db, there are however 10 entries. I've checked my server mail logs and everything, the mails aren't being blocked. I don't know whether they were sent in the first place though.
I do get other PHP notices in the logs though. Here are two (I have set the site to use MySQLi BTW):

8192,mysql_connect(): The mysql extension is deprecated and will be removed in the future: use mysqli or PDO instead,/var/www/vhosts/xxx/libraries/joomla/database/driver/mysql.php,89

8,Undefined index: rowid,/var/www/vhosts/xxx/components/com_fabrik/router.php,231

Don't know if that has anything to do with anything. This all worked 100% fine pre-J!3.6/F3.5.

I get different errors on my other sites. It's all very confusing! Most of it is copy and paste code, nothing different or fancy from site to site, except on my other sites, I use an article template.

 

[cheesegrits]

I've gone as far as I can go with Community support.

As I said in my previous message, it's almost certainly because you aren't pre-filtering the tables, so the plugin is reading the entire table.

You should figure out how to apply your conditions as pre-filters, on a copy of the list.

Nothing changed in the cron handling between 3.4 and 3.5. It's almost certainly just that your tables are growing, and you've finally hit the processing limit.

-- hugh

 

[JackGoa]

I've got like 100 entries being processed? Have a php cron that tidies things up every day.

How do I make a live copy of a list?

I honestly don't understand anything about this pre-filter thing? Can you please point me in a direction so I can figure it out?

Thanks for the help Hugh. Will contribute again once I have some funds. It's on the budget list.

bj?rn
- from tapatalk

 

[cheesegrits]

Nothing to point you at really. You just have to replace your condition code with pre-filters (List settings, under the Data tab) that do the same thing. So for instance, part of your condition is "{ahsa_leads___followed_up_raw}'=='no'", so you would add a pre-filter to the list that does ...

WHERE:
Field: followed_up
Condition: EQUALS
Value: no
Type: text

Then add other pre-filters to duplicate what the rest of your conditions are doing, joining them with "AND"

You can test them by displaying the list on the front end, and making sure it then only shows you what you expect to see.

-- hugh

 

[shawnmcginn]

Going to join in here on the conversation. I too am having an issue with scheduled tasks which I believe is similar in behavior to what you've described here. My problem is that the scheduler is only executing the condition against the first 10 records as they would be viewed in my list view. So from what I've read here, are you saying that I have to generate a second list view with a different pre-filter just for the scheduler? Does this number '10' come from the value I have in Rows per page under Details>Navigation?

 

[cheesegrits]

No, the pagination limits on the list settings shouldn't affect the cron plugin. The only limit we place is with the "Row Limit" setting in the cron connection settings (between List and Plugin). I just noticed a typo in the XML that meant the tooltip wasn't showing. That's fixed in github. It reads:

By default Fabrik will select all rows in the selected list for cron processing, subject to any pre-filters, but NOT subject to any pagination limits. This can result in memory and execution time problems on larger tables. To place a hard limit on the number of rows selected for processing (ordered as specified in your list setting), set this Row Limit.

I'm assuming you are running a relatively recent version of Fabrik, ie. at least 3.4.x. I think we fixed an issue with pagination affecting cron plugins about a year ago,

-- hugh

 

[shawnmcginn]

ok I'm running 3.4 but I don't have a row limit option in my cron plugin so it's definitely in an upgrade. Aside from that option I did create a second list as per your suggestion above which works for my application. Thanks!