Reading from and Extracting to CSV in FF

Support for iMacros. The iMacros software is the unique solution for automating every activity inside a web browser, for data extraction and web testing.
Forum rules
Before asking a question or reporting an issue:
1. Please review the list of FAQ's.
2. Use the search box (at the top of each forum page) to see if a similar problem or question has already been addressed.
3. Try searching the iMacros Wiki - it contains the complete iMacros reference as well as plenty of samples and tutorials.
4. We can respond much faster to your posts if you include the following information: CLICK HERE FOR IMPORTANT INFORMATION TO INCLUDE IN YOUR POST
Post Reply
BLiTzKrIeGx1337
Posts: 31
Joined: Sat Nov 16, 2019 10:05 pm

Reading from and Extracting to CSV in FF

Post by BLiTzKrIeGx1337 » Mon May 03, 2021 3:20 pm

VERSION BUILD=10101485
Win10
FF 88.0 (64-bit)

Good morning!

I've got a huge project I'm working on with the sales site I run, the first part of it is simple, I'm searching and plugging in data from a CSV, the part I'm having trouble with is extracting data in product pages to the CSV. All the examples and forum posts I have been able to find about extracting to a CSV are only saving to a CSV, I need to save to a CSV but also read from a CSV. Here is the script I have been testing with:

Code: Select all

VERSION BUILD=10101485
SET !DATASOURCE C:\Users\forms\Desktop\extracttest.csv
SET !LOOP 1
SET !DATASOURCE_LINE {{!LOOP}}
SET !TIMEOUT_STEP 100
URL GOTO=https://heraldpg.myprintdesk.net/DSF/Admin/MD/base.aspx#/products
TAG POS=1 TYPE=INPUT:TEXT FORM=ID:aspnetForm ATTR=ID:orderId_IntegerTxt CONTENT={{!COL1}}
Wait seconds=.5
TAG POS=1 TYPE=SPAN ATTR=TXT:{{!COL2}}
TAG POS=2 TYPE=SPAN ATTR=TXT:Pricing
TAG POS=1 TYPE=INPUT:TEXT FORM=ID:aspnetForm ATTR=ID:tbl_0_PriceCatalog_rngend_1 Extract=TXT
SET !EXTRACT {{!COL3}}
TAG POS=1 TYPE=INPUT:TEXT FORM=ID:aspnetForm ATTR=ID:tbl_0_PriceCatalog_rngend_2 Extract=TXT
SET !EXTRACT {{!COL4}}
TAG POS=1 TYPE=INPUT:TEXT FORM=ID:aspnetForm ATTR=ID:tbl_0_PriceCatalog_regularprice_1 Extract=TXT
SET !EXTRACT {{!COL10}}
TAG POS=1 TYPE=INPUT:TEXT FORM=ID:aspnetForm ATTR=ID:tbl_0_PriceCatalog_regularprice_2 Extract=TXT
SET !EXTRACT {{!COL11}}
TAG POS=1 TYPE=INPUT:TEXT FORM=ID:aspnetForm ATTR=ID:tbl_0_PriceCatalog_regularprice_3 Extract=TXT
SET !EXTRACT {{!COL12}}
SAVEAS TYPE=EXTRACT FOLDER=C:\Users\forms\desktop FILE=extracttest.csv
TAG POS=1 TYPE=INPUT:SUBMIT FORM=ID:aspnetForm ATTR=ID:ctl00_ctl00_C_M_ctl00_W_StartNavigationTemplateContainerID_ctl00_BtnSaveAndExit
It doesn't necessarily have to be the same csv file if that is one of the problems. Also, what I am trying to extract is a table, I'm not trying to extract it as a table though because when I have tried that it doesn't paste any of the correct information into the CSV. The areas for price and quantity or text areas which I think is the reason its not grabbing the values in those fields. If there is a way around that please let me know.

When I run the script above I get the error:
RuntimeError: Column number 3 greater than total number of columns 2, line: 12

Which I suspect I'm getting because the macro is still paying attention to the datasource, but I'm not sure how to tell it to ignore that.

Thank you in advance!
chivracq
Posts: 9753
Joined: Sat Apr 13, 2013 1:07 pm
Location: Amsterdam (NL)

Re: Reading from and Extracting to CSV in FF

Post by chivracq » Mon May 03, 2021 7:51 pm

BLiTzKrIeGx1337 wrote:
Mon May 03, 2021 3:20 pm

Code: Select all

VERSION BUILD=10101485
Win10
FF 88.0 (64-bit)
Good morning!

I've got a huge project I'm working on with the sales site I run, the first part of it is simple, I'm searching and plugging in data from a CSV, the part I'm having trouble with is extracting data in product pages to the CSV. All the examples and forum posts I have been able to find about extracting to a CSV are only saving to a CSV, I need to save to a CSV but also read from a CSV. Here is the script I have been testing with:

Code: Select all

VERSION BUILD=10101485
SET !DATASOURCE C:\Users\forms\Desktop\extracttest.csv
SET !LOOP 1
SET !DATASOURCE_LINE {{!LOOP}}
SET !TIMEOUT_STEP 100
URL GOTO=https://heraldpg.myprintdesk.net/DSF/Admin/MD/base.aspx#/products
TAG POS=1 TYPE=INPUT:TEXT FORM=ID:aspnetForm ATTR=ID:orderId_IntegerTxt CONTENT={{!COL1}}
Wait seconds=.5
TAG POS=1 TYPE=SPAN ATTR=TXT:{{!COL2}}
TAG POS=2 TYPE=SPAN ATTR=TXT:Pricing
TAG POS=1 TYPE=INPUT:TEXT FORM=ID:aspnetForm ATTR=ID:tbl_0_PriceCatalog_rngend_1 Extract=TXT
SET !EXTRACT {{!COL3}}
TAG POS=1 TYPE=INPUT:TEXT FORM=ID:aspnetForm ATTR=ID:tbl_0_PriceCatalog_rngend_2 Extract=TXT
SET !EXTRACT {{!COL4}}
TAG POS=1 TYPE=INPUT:TEXT FORM=ID:aspnetForm ATTR=ID:tbl_0_PriceCatalog_regularprice_1 Extract=TXT
SET !EXTRACT {{!COL10}}
TAG POS=1 TYPE=INPUT:TEXT FORM=ID:aspnetForm ATTR=ID:tbl_0_PriceCatalog_regularprice_2 Extract=TXT
SET !EXTRACT {{!COL11}}
TAG POS=1 TYPE=INPUT:TEXT FORM=ID:aspnetForm ATTR=ID:tbl_0_PriceCatalog_regularprice_3 Extract=TXT
SET !EXTRACT {{!COL12}}
SAVEAS TYPE=EXTRACT FOLDER=C:\Users\forms\desktop FILE=extracttest.csv
TAG POS=1 TYPE=INPUT:SUBMIT FORM=ID:aspnetForm ATTR=ID:ctl00_ctl00_C_M_ctl00_W_StartNavigationTemplateContainerID_ctl00_BtnSaveAndExit
It doesn't necessarily have to be the same csv file if that is one of the problems. Also, what I am trying to extract is a table, I'm not trying to extract it as a table though because when I have tried that it doesn't paste any of the correct information into the CSV. The areas for price and quantity or text areas which I think is the reason its not grabbing the values in those fields. If there is a way around that please let me know.

When I run the script above I get the error:
RuntimeError: Column number 3 greater than total number of columns 2, line: 12

Which I suspect I'm getting because the macro is still paying attention to the datasource, but I'm not sure how to tell it to ignore that.

Thank you in advance!

Hum, your FCI looks a bit "strange"..., as iMacros v10.1.0 is a Version for CR, and not for FF... :?
"Current" Version for FF88 would be v10.0.2, I would think...

Hum, alright, you do "stg" a bit wrong, those Lines..., that I have separated in 5 mini-Blocks...:

Code: Select all

TAG POS=1 TYPE=INPUT:TEXT FORM=ID:aspnetForm ATTR=ID:tbl_0_PriceCatalog_rngend_1 Extract=TXT
SET !EXTRACT {{!COL3}}
'>
TAG POS=1 TYPE=INPUT:TEXT FORM=ID:aspnetForm ATTR=ID:tbl_0_PriceCatalog_rngend_2 Extract=TXT
SET !EXTRACT {{!COL4}}
'>
TAG POS=1 TYPE=INPUT:TEXT FORM=ID:aspnetForm ATTR=ID:tbl_0_PriceCatalog_regularprice_1 Extract=TXT
SET !EXTRACT {{!COL10}}
'>
TAG POS=1 TYPE=INPUT:TEXT FORM=ID:aspnetForm ATTR=ID:tbl_0_PriceCatalog_regularprice_2 Extract=TXT
SET !EXTRACT {{!COL11}}
'>
TAG POS=1 TYPE=INPUT:TEXT FORM=ID:aspnetForm ATTR=ID:tbl_0_PriceCatalog_regularprice_3 Extract=TXT
SET !EXTRACT {{!COL12}}
... look very strange...! 5 times you extract some Data from the Site, ... to each time directly discard that 'EXTRACT' and replace it directly with some Content from the DataSource...
=> In the end, the 'SAVEAS' Command will simply "resave" the Content of 'COL_12' (if it exists, indeed, ah-ah...!), by appending a New Row to the '.CSV' File and saving that Content to 'COL_1'. And there is only one "fake" 'EXTRACT', so 'SAVEAS' only saves to 1 Col.

Maybe you mean to use "ADD !EXTRACT" (x5 times), instead of "SET !EXTRACT", then the 'SAVEAS' would save your 5 'EXTRACT's + the 5 existing Cols into 10 Cols in total... :idea:
- (F)CI(M) = (Full) Config Info (Missing): iMacros + Browser + OS (+ all 3 Versions + 'Free'/'PE').
- I don't even read the Qt if that (required) Info is not mentioned...!
- Script & URL help a lot for more "educated" Help...
BLiTzKrIeGx1337
Posts: 31
Joined: Sat Nov 16, 2019 10:05 pm

Re: Reading from and Extracting to CSV in FF

Post by BLiTzKrIeGx1337 » Fri May 07, 2021 6:35 pm

Hey chivracq, I appreciate the help, sorry for the late reply. For the record, I wasn't getting "ADD !EXTRACT" to work but it turns out it is currently unsupported in FF, "SET !EXTRACTADD" still works though. iMacros Wiki link

And actually, after reading into into it on the wiki I don't think I'm using the right method for what I'm trying to do, !EXTRACTADD is for adding an extracted value to an existing value in the designated column. I just want to extract a value and add it to a new column that I designate, how could I do that?

Just to be clear, I've got a csv sheet that looks like this:
4287 | ACI015 Appleton Fee Inserts
4298 | ACI019 Albertville Letterhead
89 | [GC] Admin Letterhead

I need whats currently there to search for what I want to extract but also to know what each of the extracted values are later on, so after extraction I want something that looks like this:
4287 | ACI015 Appleton Fee Inserts |100|200|10|17
4298 | ACI019 Albertville Letterhead |100|200|15|22
89 | [GC] Admin Letterhead |100|200|11|14
chivracq
Posts: 9753
Joined: Sat Apr 13, 2013 1:07 pm
Location: Amsterdam (NL)

Re: Reading from and Extracting to CSV in FF

Post by chivracq » Fri May 07, 2021 7:18 pm

BLiTzKrIeGx1337 wrote:
Fri May 07, 2021 6:35 pm
Hey chivracq, I appreciate the help, sorry for the late reply. For the record, I wasn't getting "ADD !EXTRACT" to work but it turns out it is currently unsupported in FF, "SET !EXTRACTADD" still works though. iMacros Wiki link

And actually, after reading into into it on the wiki I don't think I'm using the right method for what I'm trying to do, !EXTRACTADD is for adding an extracted value to an existing value in the designated column. I just want to extract a value and add it to a new column that I designate, how could I do that?

Just to be clear, I've got a csv sheet that looks like this:

Code: Select all

4287	| ACI015 Appleton Fee Inserts 
4298	| ACI019 Albertville Letterhead 
89	| [GC] Admin Letterhead 
I need whats currently there to search for what I want to extract but also to know what each of the extracted values are later on, so after extraction I want something that looks like this:

Code: Select all

4287	| ACI015 Appleton Fee Inserts   |100|200|10|17
4298	| ACI019 Albertville Letterhead |100|200|15|22
89	| [GC] Admin Letterhead          |100|200|11|14

Oh...!, yep indeed, nice to get some Follow-up, I sent a few "Reminders" yesterday in a few Threads, and "you" were going to be next later on today, ah-ah...! :wink:

Hum..., "ADD !EXTRACT not being supported in FF" (=> with iMacros for FF v10.0.2, I reckon, as I mentioned that your FCI was not completely "accurate") sounds a bit "strange" to me, I don't know about that, and that's not mentioned in the Wiki/Release Notes for v10.0.2 for FF...! :o

But "funny" to hear that "SET !EXTRACTADD" still works in that Version, I haven't used/seen it in years, now, ah-ah...!, but it used to work exactly like the 'ADD' Command...

Stg possible about "ADD !EXTRACT" is that "many Users" use the 'ADD' Command a bit "incorrectly", for example "ADD !VAR1 xxx" will return some Error if '!VAR1' has not been used/defined earlier in the Script.
And only the '!EXTRACT' Built-in Var was "tolerant" enough, that "ADD !EXTRACT xxx" would be "tolerated"/accepted by iMacros, even if the Script didn't contain any 'EXTRACT' (yet), which would be your Case, as you want to "start" with your 2 existing Cols, before adding the 4 'EXTRACT's in Cols [3-6].

=> The "proper" Way, is to first do a "SET !EXTRACT" the "first time", and only after to use the 'ADD' Command...:

Code: Select all

SET !EXTRACT {{!COL1}}
ADD !EXTRACT {{!COL2}}

'Do the 4x 'TAG ... EXTRACT=TXT

SAVEAS TYPE=EXTRACT etc...
Another Way is to first do the 4x 'EXTRACT's, and to reconstruct the Content of the '!EXTRACT' Var "at the end" like you want to order it in the '.CSV' like you want:

Code: Select all

'Do the 4x 'TAG ... EXTRACT=TXT

'Prepare Data for the 'SAVEAS':
SET EXTRACT_Temp {{!EXTRACT}}
SET !EXTRACT {{!COL1}}
ADD !EXTRACT {{!COL2}}
ADD !EXTRACT {{EXTRACT_Temp}}

SAVEAS TYPE=EXTRACT etc...
... which can be a bit shortened, if you want:

Code: Select all

'Do the 4x 'TAG ... EXTRACT=TXT

'Prepare Data for the 'SAVEAS':
SET EXTRACT_Temp {{!EXTRACT}}
SET !EXTRACT {{!COL1}}[EXTRACT]{{!COL2}}[EXTRACT]{{EXTRACT_Temp}}

SAVEAS TYPE=EXTRACT etc...
... as the 'SAVEAS' Command uses the "[EXTRACT]" String as a "Separator" in the '!EXTRACT' Var to re-split all 'EXTRACT's into separate Columns...
Still following...?, ah-ah...!! :wink:
- (F)CI(M) = (Full) Config Info (Missing): iMacros + Browser + OS (+ all 3 Versions + 'Free'/'PE').
- I don't even read the Qt if that (required) Info is not mentioned...!
- Script & URL help a lot for more "educated" Help...
BLiTzKrIeGx1337
Posts: 31
Joined: Sat Nov 16, 2019 10:05 pm

Re: Reading from and Extracting to CSV in FF

Post by BLiTzKrIeGx1337 » Fri May 07, 2021 8:24 pm

Yes I follow, so this works -

Code: Select all

VERSION BUILD=10101485
SET !DATASOURCE C:\Users\forms\Desktop\extracttest.csv
SET !LOOP 1
SET !DATASOURCE_LINE {{!LOOP}}
SET !TIMEOUT_STEP 10
URL GOTO=https://heraldpg.myprintdesk.net/DSF/Admin/MD/base.aspx#/products
TAG POS=1 TYPE=INPUT:TEXT FORM=ID:aspnetForm ATTR=ID:orderId_IntegerTxt CONTENT={{!COL1}}
Wait seconds=.5
TAG POS=1 TYPE=SPAN ATTR=TXT:{{!COL2}}
TAG POS=2 TYPE=SPAN ATTR=TXT:Pricing
SET !EXTRACT {{!COL3}}
ADD !EXTRACT {{!COL4}}
TAG POS=1 TYPE=INPUT:TEXT FORM=ID:aspnetForm ATTR=ID:tbl_0_PriceCatalog_rngend_1 Extract=TXT
TAG POS=1 TYPE=INPUT:TEXT FORM=ID:aspnetForm ATTR=ID:tbl_0_PriceCatalog_rngend_2 Extract=TXT
TAG POS=1 TYPE=INPUT:TEXT FORM=ID:aspnetForm ATTR=ID:tbl_0_PriceCatalog_regularprice_1 Extract=TXT
TAG POS=1 TYPE=INPUT:TEXT FORM=ID:aspnetForm ATTR=ID:tbl_0_PriceCatalog_regularprice_2 Extract=TXT
TAG POS=1 TYPE=INPUT:TEXT FORM=ID:aspnetForm ATTR=ID:tbl_0_PriceCatalog_regularprice_3 Extract=TXT
SAVEAS TYPE=EXTRACT FOLDER=C:\Users\forms\desktop FILE=extracttest.csv
TAG POS=1 TYPE=INPUT:SUBMIT FORM=ID:aspnetForm ATTR=ID:ctl00_ctl00_C_M_ctl00_W_StartNavigationTemplateContainerID_ctl00_BtnSaveAndExit
The only issue(s) is (I think the 2 issues here might be related) it throws "RuntimeError: Column number 3 greater than total number of columns 2, line: 11". I tried to cheese it by just putting spaces in the first row for the next few columns. But instead of putting the extracted values in Col3, 4, 5, etc. it puts them in col1, 2, 3, etc. starting in a new row below my last item number and name. Like this:
4287 ACI015 Appleton Fee Inserts
4298 ACI019 Albertville Letterhead
89 [GC] Admin Letterhead
50 100 0.4226 0.2521 0.0817
250 500 0.772 0.444 0.255
chivracq
Posts: 9753
Joined: Sat Apr 13, 2013 1:07 pm
Location: Amsterdam (NL)

Re: Reading from and Extracting to CSV in FF

Post by chivracq » Sat May 08, 2021 11:03 pm

BLiTzKrIeGx1337 wrote:
Fri May 07, 2021 8:24 pm
Yes I follow, so this works -

Code: Select all

VERSION BUILD=10101485
SET !DATASOURCE C:\Users\forms\Desktop\extracttest.csv
SET !LOOP 1
SET !DATASOURCE_LINE {{!LOOP}}
SET !TIMEOUT_STEP 10
URL GOTO=https://heraldpg.myprintdesk.net/DSF/Admin/MD/base.aspx#/products
TAG POS=1 TYPE=INPUT:TEXT FORM=ID:aspnetForm ATTR=ID:orderId_IntegerTxt CONTENT={{!COL1}}
Wait seconds=.5
TAG POS=1 TYPE=SPAN ATTR=TXT:{{!COL2}}
TAG POS=2 TYPE=SPAN ATTR=TXT:Pricing
SET !EXTRACT {{!COL3}}
ADD !EXTRACT {{!COL4}}
TAG POS=1 TYPE=INPUT:TEXT FORM=ID:aspnetForm ATTR=ID:tbl_0_PriceCatalog_rngend_1 Extract=TXT
TAG POS=1 TYPE=INPUT:TEXT FORM=ID:aspnetForm ATTR=ID:tbl_0_PriceCatalog_rngend_2 Extract=TXT
TAG POS=1 TYPE=INPUT:TEXT FORM=ID:aspnetForm ATTR=ID:tbl_0_PriceCatalog_regularprice_1 Extract=TXT
TAG POS=1 TYPE=INPUT:TEXT FORM=ID:aspnetForm ATTR=ID:tbl_0_PriceCatalog_regularprice_2 Extract=TXT
TAG POS=1 TYPE=INPUT:TEXT FORM=ID:aspnetForm ATTR=ID:tbl_0_PriceCatalog_regularprice_3 Extract=TXT
SAVEAS TYPE=EXTRACT FOLDER=C:\Users\forms\desktop FILE=extracttest.csv
TAG POS=1 TYPE=INPUT:SUBMIT FORM=ID:aspnetForm ATTR=ID:ctl00_ctl00_C_M_ctl00_W_StartNavigationTemplateContainerID_ctl00_BtnSaveAndExit

Ah...!, OK, so the "SET/ADD !EXTRACT" Func does work "after all" for you, ah-ah...! You "scared" me a little bit, but OK, good...! :D

>>>
BLiTzKrIeGx1337 wrote:
Fri May 07, 2021 8:24 pm
The only issue(s) is (I think the 2 issues here might be related) it throws "RuntimeError: Column number 3 greater than total number of columns 2, line: 11". I tried to cheese it by just putting spaces in the first row for the next few columns. But instead of putting the extracted values in Col3, 4, 5, etc. it puts them in col1, 2, 3, etc. starting in a new row below my last item number and name. Like this:

Code: Select all

4287	ACI015 Appleton Fee Inserts 	 	 	 
4298	ACI019 Albertville Letterhead 			
89	[GC] Admin Letterhead 			
50	100	0.4226	0.2521	0.0817
250	500	0.772	0.444	0.255

Yeah, but for that part, I'm asking you for the 3rd time to clarify your FCI, please...!? :!: :?:

All v10.x.x Versions (for FF + CR) are super-Buggy, but with different Bugs or "Behaviours"... :P , even if they are all/both based on the same Framework, but I need to know exactly which Version/which FCI you are using...!
Because you keep mentioning FF as your Browser, but you mentioned a Version in your FCI in your OP which is a Version for CR (only!) and not for FF, and your Script shows also that same v10.1.0 Version (for CR, and not for FF)... :?
(And edit/update your OP with the "correct" Info as well...)

Hum..., and I will also need to know in which Software/Program do you create and edit your '.CSV' File...?
=> 'Notepad'...?, or 'Excel' or 'OO'/'LO' Equivalent...? (+ the Version maybe...)
('OO'/'LO' = 'Open Office' / 'Libre Office')
- (F)CI(M) = (Full) Config Info (Missing): iMacros + Browser + OS (+ all 3 Versions + 'Free'/'PE').
- I don't even read the Qt if that (required) Info is not mentioned...!
- Script & URL help a lot for more "educated" Help...
BLiTzKrIeGx1337
Posts: 31
Joined: Sat Nov 16, 2019 10:05 pm

Re: Reading from and Extracting to CSV in FF

Post by BLiTzKrIeGx1337 » Tue May 11, 2021 5:32 pm

My version of excel is Microsoft 365 MSO (16.0.13929.20222) 64-bit.

I'm confused about the FCI you keep saying I need to fix, recently my iMacros auto updated (which didn't really make sense at the time based on the release notes page it took me to).
I just checked one of my older macros and the version it has is 10021450, so I'd guess thats the version that it updated me out of? If your also saying that makes no sense, maybe I should just reinstall iMacros (?), I haven't really noticed any issues, only the I noticed that changed was the color scheme of the iMacros window, color scheme was green/gray/white and now its blue/gray/white.
I also double checked my FF version, it says its up to date - 88.0.1 (64-bit).

Also, quick side question I was trying to find a solution to this morning if you don't mind. The site I used most of my macros on uses cookies to detect multiple active sessions running at once and will log you out if it is detected. So naturally I figured I could run more macros in private browsing windows, but despite insuring that iMacros is enabled for Private Browsing windows, the iMacros icon is grayed out and on hover it shows the tooltip "Private Browsing: iMacros disabled due to Firefox restrictions.", any ideas on what else I could try enabling/other solutions?

Thanks!
Post Reply