• Hello Fabrik Community

    Fabrik is now in the hands of the development team that brought you Fabrik for Joomla 4. We have recently transitioned the Fabrik site over to a new server and are busy trying to clean it up. We have upgraded the site to Joomla 4 and are running the latest version of Fabrik 4. We have also upgraded the Xenforo forum software to the latest version. Many of the widgets you might have been used to on the forum are no longer operational, many abandoned by the developers. We hope to bring back some of the important ones as we have time.

    Exciting times to be sure.

    The Fabrik 4.0 Official release is now available. In addition, the Fabrik codebase is now available in a public repository. See the notices about these in the announcements section

    We wish to shout out a very big Thank You to all of you who have made donations. They have really helped. But we can always use more...wink..wink..

    Also a big Thank You to those of you who have been assisting others in the forum. This takes a very big burden off of us as we work on bugs, the website and the future of Fabrik.

listcsv plugin

ghicar

Member
I make use of list plugin "listcsv" to run some php code when importing from CSV. I see the plugin is listed as planned for inclusion for Fabrik4.0. It would be great to see this available soon for beta testing.
 
listcsv.zip
We didn't test it until now but it is at least installable.
So if you want to try...
 

Attachments

  • listcsv.zip
    17.1 KB · Views: 68
troester, thanks for the zip, noted, its untested so thought I would give it a try without any application php code applied in the plugin usage first, and get this error:

Call to a member function canRepeat() on null
Call stack
# Function Location
1 () JROOT/components/com_fabrik/models/importcsv.php:1204
2 FabrikFEModelImportcsv->_fakeJoinData() JROOT/components/com_fabrik/models/importcsv.php:921
3 FabrikFEModelImportcsv->insertData() JROOT/components/com_fabrik/controllers/import.php:115
4 FabrikControllerImport->doimport() JROOT/libraries/src/MVC/Controller/BaseController.php:672
5 Joomla\CMS\MVC\Controller\BaseController->execute() JROOT/components/com_fabrik/fabrik.php:200
6 require_once() JROOT/libraries/src/Dispatcher/LegacyComponentDispatcher.php:71
7 Joomla\CMS\Dispatcher\LegacyComponentDispatcher::Joomla\CMS\Dispatcher\{closure}() JROOT/libraries/src/Dispatcher/LegacyComponentDispatcher.php:73
8 Joomla\CMS\Dispatcher\LegacyComponentDispatcher->dispatch() JROOT/libraries/src/Component/ComponentHelper.php:355
9 Joomla\CMS\Component\ComponentHelper::renderComponent() JROOT/libraries/src/Application/SiteApplication.php:200
10 Joomla\CMS\Application\SiteApplication->dispatch() JROOT/libraries/src/Application/SiteApplication.php:241
11 Joomla\CMS\Application\SiteApplication->doExecute() JROOT/libraries/src/Application/CMSApplication.php:294
12 Joomla\CMS\Application\CMSApplication->execute() JROOT/includes/app.php:61
13 require_once() JROOT/index.php:32

I tried putting a null check on the call at /com_fabrik/models/importcsv.php:1204

if (!is_null( $groups[$join->group_id] ))
{
$repeat = $groups[$join->group_id]->canRepeat();
}
else
{
$repeat = NULL;
}


but it then fails later in the code:

count(): Argument #1 ($value) must be of type Countable|array, int given
Call stack
# Function Location
1 () JROOT/components/com_fabrik/models/importcsv.php:1128
2 FabrikFEModelImportcsv->insertJoinedData() JROOT/components/com_fabrik/models/importcsv.php:927
3 FabrikFEModelImportcsv->insertData() JROOT/components/com_fabrik/controllers/import.php:115
4 FabrikControllerImport->doimport() JROOT/libraries/src/MVC/Controller/BaseController.php:672
5 Joomla\CMS\MVC\Controller\BaseController->execute() JROOT/components/com_fabrik/fabrik.php:200
6 require_once() JROOT/libraries/src/Dispatcher/LegacyComponentDispatcher.php:71
7 Joomla\CMS\Dispatcher\LegacyComponentDispatcher::Joomla\CMS\Dispatcher\{closure}() JROOT/libraries/src/Dispatcher/LegacyComponentDispatcher.php:73
8 Joomla\CMS\Dispatcher\LegacyComponentDispatcher->dispatch() JROOT/libraries/src/Component/ComponentHelper.php:355
9 Joomla\CMS\Component\ComponentHelper::renderComponent() JROOT/libraries/src/Application/SiteApplication.php:200
10 Joomla\CMS\Application\SiteApplication->dispatch() JROOT/libraries/src/Application/SiteApplication.php:241
11 Joomla\CMS\Application\SiteApplication->doExecute() JROOT/libraries/src/Application/CMSApplication.php:294
12 Joomla\CMS\Application\CMSApplication->execute() JROOT/includes/app.php:61
13 require_once() JROOT/index.php:32
 
This is running on joined lists. Do you have a list join?
Do you have data in your csv file?
Is it importing correctly whithout listcsv plugin enabled?

Do you have a Fabrik3 but php8 test case somewhere?

PHP8 needs a lot of these Countable, array to string etc. fixed (so not necessarily a Fabrik4 issue)

BTW: Did you see
You are invited to join other community members active in coding, maintaining and improving Fabrik. Please visit https://fabrik.help for more information!
 
Re the questions:
Do you have a list join?
Yes, the list has several elements that join to the Joomla users table.
Do you have data in your csv file?
The test CSV file I was using has two rows of test data, the test import works, but the extra php added via the listcsv is not running. When I run the import on my J3 site running Fabrik 3.10 and PHP8 It works fully.
Is it importing correctly without listcsv plugin enabled?
The test data is importing correctly into the list with or without the plugin applied. However, the php code that should be run "After Import Row PHP Code" is not being run. And the exception being thrown happens with or without php code assigned to be run at the "After Import Row" stage.​

Do you have a Fabrik3 but php8 test case somewhere?
I do have a site on J3 running Fabrik 3.10 and PHP8.1 where the listcsv plugin is working. It does report an error:
Debug: Caught exception on eval in onAfterImportCSVRow : auto_detect_line_endings is deprecated
But the import works and the added php script is doing what is expected despite this error.​

I will do further debugging.
 
Apologies, on previous answer, the test data does NOT import into the table on J4 without the listcsv applied. So the issue lies with basic CSV import. I have been doing some comparisons of behaviour in J3 with Fabrik3.10 and J4 with Fabrik 4beta4b zip. On these tests I have not yet installed listcsv plugin on J4.

I have found in /components/com_fabrik/models/element.php there is a function onStoreRow which loads a FabrikFEModeForm variable $formModel = $this->getFormModel();

On J4 using dd() (dump&die function) I can see there are two keys formData & formDataWithTableName. When I look at the contents, in formData I see the element values from the CSV file but when I look in formDataWithTableNames it does not contain any of the elements being imported. Unfortunately, the code is using data from formDataWithTableName so nothing gets imported.
When I compare this with J3/F3.10 both of these keys have the imported data within:
See uploaded screen shot of J4 (Screenshot-J4-formModelVar.png) log extract for J3 (LogExtract-J3-formModelVar.txt) - I could not get dd() working on J3 for some reason.
It also seems the J3 has a set of numbered dummy elements that are not used, I dont know where they come from
It also seems J4 has an id and an id_raw in formDataWithTableName which are not present in J3 log output.

If anyone has any pointers on where the problem lies please advise, or if indeed something has already been fixed here.
 

Attachments

  • Screenshot-J4-formModelVar.png
    Screenshot-J4-formModelVar.png
    21.1 KB · Views: 49
  • LogExtract-J3-formModelVar.txt
    2.8 KB · Views: 46
Looking at csv import code paths for J3/F3.10 and J4/F4beta4b, they seem to deviate in determination of whether or not special treatment for joins is required.

in /components/com_fabrik/models/importcsv.php, there is a function tableJoinsFound() which iterates the model joins looking for one that matches this condition:

if ((int) $joins[$x]->list_id !== 0 && $joins[$x]->element_id === 0)

in J3/F3.10 it iterates an array with entries like this
array(11) {
[0]=>
object(stdClass)#1177 (13) {
["id"]=>
string(3) "108"
["list_id"]=>
string(2) "11"
["element_id"]=>
string(1) "0"



In J4/F4beta4b the same list has this:
array(7) {
[0]=>
object(stdClass)#1291 (13) {
["id"]=>
int(108)
["list_id"]=>
int(11)
["element_id"]=>
int(0)

Note the fields that have switched from string to int, which makes the above if statement pass in J4/F4beta4b where it does not pass in J3/F3.10 due to type string.

As the single db join in the test list that I am trying csv imports to is not relevent for the import, I temporarily changed (hacked) this function to always return false ie "no joins found", and the CSV import worked!

I then installed the listcsv plugin and applied to the list and tried the import again and the plugin worked as expected.

Appreciate what I have done is not a good fix. If joins are found then the codepath goes through a function _fakeJoinData to build the form fields for import, which is where the fix probably needs to go, but then again, the J3/F3.10 is not going through that function.
 
Last edited:
Ok, so it seems it's a "bug" in F3 (doing no (int) on the element_id, so never running this condition) which was "fixed" in F4 and then failing the import.

So the listcsv plugin is running as it is if the general import in .../components/com_fabrik/models/importcsv.php is fixed/hacked?
 
Yes troester, summary is about right. I have done some further debugging and this is what I have found.
When no joins found, the code copies the CSV data into two array locations via $formModel->formData = $formModel->formDataWithTableName = $aRow;
Whereas when joins were found it was only copying to one location: $formModel->formData = $data;
From the comments I found in the code, it looks like the dev team were in the process of switching from formDataWithTableName to formData.
I changed the joineddatafound code path to copy to both locations, and that resulted in one row being imported (the last row in the CSV file).

Next problem: Looking at the code, for joineddatafound, it only inserts once outside of the loop round the rows of CSV. So I moved that inside and it inserted all rows.

Next problem: However, the listcsv plugin was not being actioned, and in the code you can see that none of the listcsv plugin events are called when following the joineddatafound code path.

That can be fixed, but I think the code needs to be tested against a CSV import that tries to import data into joined tables.

Oh also, Applying the (int) cast fix for J3/F3.10, causes the PHP exceptions I mentioned at the beginning of this topic around canRepeat countable.
 
Putting the listcsv plugin to one side and focusing on CSV imports, they seem quite broken in both J!3 & J!4 when the list has joins. I setup a very simple test case of two joined tables.

As discussed earlier in this thread, in J!3 a missing type cast means the import logic does not recognise there are table joins involved and does work to some degree in that it imports into the primary table ok, but not the joined table. On J!4 the join is detected but the logic fails to import anything correctly even into the primary table. Also if J!3 is fixed to detect the join, it also fails in the same way as J!4. There were a few PHP8 errors that have to be fixed when going through the code path when joins are detected.

I don't know if going back to J!3 & PHP7 would result in it working as I don't currently have an environment to set that up in. Not sure if anyone has csv imports working when table joins are involved?

I don't actually even know what the data should officially look like in the CSV file for the table joins. What I do know is that for CSV exports the output structure depends on how the repeatable groups are configured, ie single row for each group or merge.

I would feel more confident about looking to fixing if I had something that worked properly on earlier versions of Joomla/PHP to use as a comparison for where it deviates.

For my website a workable solution is to import the joined table separately. Ie import the main table, and export to get primary keys, then import the data into the joined table using the new foreign keys. That works well on J3 but is broken in J!4 due to the table join being detected. You can get round that in J!4 by either forcing it to ignore the joins (in the code), or creating a new list for the same table but exclude the join and import into that list instead.

It would be good to know if anyone has had CSV imports with table joins involved and what versions of Joomla and PHP it used.
 
I am still finding CSV import very broken when the table has joins. As discussed on this thread, in J!3 the import code did not detect joins and just imported into the main table.

In J!4 for fabric lists with a join, I first get a php count error, which I fixed with some code applied around line 1296 of ~/components/com_fabrik/models/form.php:

Code:
                        $smallerElHTMLName = $tmpElement->getFullName(true, false);
                        $target = $this->formData[$smallerElHTMLName];

                        // mimick php 7 behaviour
                        if(is_null($target))
                        {
                            $repeatGroup = 0;
                        }
                        else if(is_countable($target))
                        {
                            $repeatGroup = count($target);
                        }
                        else
                        {
                            $repeatGroup = 1;
                        }
                        //$repeatGroup       = count($this->formData[$smallerElHTMLName]);

But I also find the import does not import anything useful, and I have had to resort to changing the code to not detect any joins with a hack around line 801 of ~/components/com_fabrik/models/import.php

Code:
        $tableJoinsFound = $this->tableJoinsFound();

        // HACK HACK, import with joins very broken so hard code no joins found
        $tableJoinsFound = false;

I have had to apply these changes through the various releases of Fabrik!4 and ask if the base code can have something similar applied until such time that CSV imports with joins are worked on and fixed.

If anyone has CSV imports to lists with joins working please advise.
 
We are in need of some funding.
More details.

Thank you.

Members online

Back
Top