Saving to and importing strings from local text file

Discussions and Tech Support related to the iMacros Firefox Add-on, including the built-in Javascript scripting interface for playing .js files.

Moderators: Community Moderators, iMacros Moderators

Forum rules
Before asking a question or reporting an issue:
1. Please review the list of FAQ's.
2. Use the Google search box (at the top of each forum page) to see if a similar problem or question has already been addressed. This will search the entire contents of the forums as well as the iMacros Wiki.
3. We can respond much faster to your posts if you include the following information:

CLICK HERE FOR IMPORTANT INFORMATION TO INCLUDE IN YOUR POST

Answering your own posts (e.g. attempting to "bump" your topic) drops your topic from the list of unanswered threads, so it may actually receive less views.

Saving to and importing strings from local text file

by microuser3 on Sat Nov 19, 2016 6:23 pm

I have been for many hours trying to figure out a way to do this in vain. I need to be able to extract strings from a site, modify them with javascript(so far no problem), but I need to be able to save this data locally, and be able to fetch it, modify it, use includes() method on the file, append new data to it, etc... I have tried seemingly everything that is in the documentation such as SAVEAS and !DATASOURCE but they all seem to have tight limitations that make my project hit a dead end.

If anyone knows of a solution to this and would be willing to help it would be greatly appreciated. I use the latest version of firefox, iMacros, Windows 7, etc..

thank you

*running:
VERSION BUILD=9030808
Windows 7 Home Premium
Firefox 50.0
Last edited by microuser3 on Sat Nov 19, 2016 8:28 pm, edited 1 time in total.
microuser3
 
Posts: 11
Joined: Sat Nov 19, 2016 6:09 pm

Re: Saving to and importing strings from local text file

by chivracq on Sat Nov 19, 2016 8:01 pm

microuser3 wrote:I have been for many hours trying to figure out a way to do this in vain. I need to be able to extract strings from a site, modify them with javascript(so far no problem), but I need to be able to save this data locally, and be able to fetch it, modify it, use includes() method on the file, append new data to it, etc... I have tried seemingly everything that is in the documentation such as SAVEAS and !DATASOURCE but they all seem to have tight limitations that make my project hit a dead end.

If anyone knows of a solution to this and would be willing to help it would be greatly appreciated. I use the
Code: Select all
latest version of firefox, iMacros, Windows 7, etc..


thank you

FCIM...! :mrgreen:
"latest version of firefox, iMacros," still gives a 16-20 Matrix of possible Configurations to choose from and won't mean anything to anybody reading the Thread in a few weeks/months/years, give the exact Versions...

From a quick read of your Qt as I won't do any "thinking" until FCI is mentioned (sorry, start using the Forum "correctly" from the beginning...), yep, '!DATASOURCE' + 'SAVEAS' (+ 'FILEDELETE' that you don't mention) is one way indeed, depending on if you use an '.iim' or '.js' Script, the Size of your File (Nb of Rows) (=> indirect Qt's...), and there are a few other Options...
+ What are those "tight limitations" that you've encountered...?
And it could be handy if you provided a short Sample of your File, with a few Rows (directly in your Thread, in some 'CODE' Field)...
No URL + No Script => Generic Advice...
- (F)CIM = (Full) Config Info Missing: iMacros + Browser + OS with all 3 Versions...
- I usually don't even read the Question if that (required) Info is not mentioned...
- Script & URL usually help a lot for a more "educated" Help...
chivracq
 
Posts: 6373
Joined: Sat Apr 13, 2013 6:07 am
Location: Amsterdam (NL)

Re: Saving to and importing strings from local text file

by microuser3 on Sat Nov 19, 2016 8:26 pm

VERSION BUILD=9030808
Windows 7 Home Premium
Firefox 50.0
microuser3
 
Posts: 11
Joined: Sat Nov 19, 2016 6:09 pm

Re: Saving to and importing strings from local text file

by microuser3 on Sat Nov 19, 2016 9:21 pm

chivracq wrote:

From a quick read of your Qt as I won't do any "thinking" until FCI is mentioned (sorry, start using the Forum "correctly" from the beginning...), yep, '!DATASOURCE' + 'SAVEAS' (+ 'FILEDELETE' that you don't mention) is one way indeed, depending on if you use an '.iim' or '.js' Script, the Size of your File (Nb of Rows) (=> indirect Qt's...), and there are a few other Options...
+ What are those "tight limitations" that you've encountered...?
And it could be handy if you provided a short Sample of your File, with a few Rows (directly in your Thread, in some 'CODE' Field)...
No URL + No Script => Generic Advice...


Ok so basically i need to build a database by adding "ID" numbers such as "242166161" extracted from web pages into local storage, they don't come right in that format so I use some javascript to get rid of whats not necessary and keep only the number I want. I then need to know if new numbers in the future have already been cataloged, and if not, add it to the list and keep working from the same list every time I run the macro. I tried putting everything in row1,col1 of a .csv, (seperated by dots) and assigning that string to a .js variable using !DATASOURCE. Problem is that when the time comes to save the data back to the same .csv file using a js variable there doesn't seen to be a way to do it.

The best would be if I could just save data on a .txt file with .js to store there whatever strings i want to keep, and be able to load that file just as easily and assign it to a variable, but from my researches this is simply not possible. But i cannot believe there isn't some kind of workaround to get it done..I'm sure many users would need that kind of function.

Heres a sample, extracting from a textfile i made in notepad
Code: Select all
iimPlay("my project/action_09.iim")
var ID = []

for (i=0;i<10;i++) {
if (iimGetLastExtract(i+1).length!=14) {
ID[i]="(000000000000)";
} else {
ID[i]=iimGetLastExtract(i+1);
}
}


var load;
load =  "CODE:";
load +=  "set !extract null" + "\n";
load +=  "SET !DATASOURCE ID.csv" + "\n";
load +=  "SET !DATASOURCE_COLUMNS 1" + "\n";
load +=  "SET !DATASOURCE_LINE 1" + "\n";
load +=  "SET !extract {{!col1}}" + "\n";
iimPlay(load);
var history=iimGetLastExtract(0);
microuser3
 
Posts: 11
Joined: Sat Nov 19, 2016 6:09 pm

Re: Saving to and importing strings from local text file

by chivracq on Sun Nov 20, 2016 10:05 am

microuser3 wrote:
Code: Select all
VERSION BUILD=9030808
Windows 7 Home Premium
Firefox 50.0

OK for FCI... :D

v9.0.3 is btw "a bit" buggy and limited compared to the much more stable v8.9.7 previous Version as you will see a bit further in my Comments in this Post... But hum, we haven't had any User(s) yet confirming that v8.9.7 (still) works (correctly!) on FF50 and that FF50 doesn't break any Command(s) as nearly every new FF Version(s) break(s) some Command(s) in iMacros for the last 20 FF Versions... :roll:

microuser3 wrote:
chivracq wrote:From a quick read of your Qt as I won't do any "thinking" until FCI is mentioned (sorry, start using the Forum "correctly" from the beginning...), yep, '!DATASOURCE' + 'SAVEAS' (+ 'FILEDELETE' that you don't mention) is one way indeed, depending on if you use an '.iim' or '.js' Script, the Size of your File (Nb of Rows) (=> indirect Qt's...), and there are a few other Options...
+ What are those "tight limitations" that you've encountered...?
And it could be handy if you provided a short Sample of your File, with a few Rows (directly in your Thread, in some 'CODE' Field)...
No URL + No Script => Generic Advice...


Ok so basically i need to build a database by adding "ID" numbers such as "242166161" extracted from web pages into local storage, they don't come right in that format so I use some javascript to get rid of whats not necessary and keep only the number I want. I then need to know if new numbers in the future have already been cataloged, and if not, add it to the list and keep working from the same list every time I run the macro. I tried putting everything in row1,col1 of a .csv, (seperated by dots) and assigning that string to a .js variable using !DATASOURCE. Problem is that when the time comes to save the data back to the same .csv file using a js variable there doesn't seen to be a way to do it.

The best would be if I could just save data on a .txt file with .js to store there whatever strings i want to keep, and be able to load that file just as easily and assign it to a variable, but from my researches this is simply not possible. But i cannot believe there isn't some kind of workaround to get it done..I'm sure many users would need that kind of function.

Heres a sample, extracting from a textfile i made in notepad
Code: Select all
iimPlay("my project/action_09.iim")
var ID = []

for (i=0;i<10;i++) {
if (iimGetLastExtract(i+1).length!=14) {
ID[i]="(000000000000)";
} else {
ID[i]=iimGetLastExtract(i+1);
}
}


var load;
load =  "CODE:";
load +=  "set !extract null" + "\n";
load +=  "SET !DATASOURCE ID.csv" + "\n";
load +=  "SET !DATASOURCE_COLUMNS 1" + "\n";
load +=  "SET !DATASOURCE_LINE 1" + "\n";
load +=  "SET !extract {{!col1}}" + "\n";
iimPlay(load);
var history=iimGetLastExtract(0);

OK, I get the Scenario, I only notice that the "modify it" part from your first Description in "... I need to be able to save this data locally, and be able to fetch it, modify it, use includes() method on the file, append new data to it, etc...", which could be a tricky part has disappeared from your second Description where you basically only want to "add" (= append) some new Data.
This is an "important" Requirement as the "rest" is fairly easy (fetch, check/compare and append) but editing the existing Content of a DataSource is more complicated, so you'd need to clarify on that...

But further, Good News...!, all the Assumptions you make about "impossible" are wrong I think, ah-ah...!
First you could have a look at the following Thread which is a bit similar to your Scenario even if the "save/add new Data to the File" Mechanism is a bit missing as it was done manually or from some other Macro, but I had already mentioned several Methods that would be relevant for your Case... (+ With concrete Implementation and Full Script (in pure '.iim').):
- exclude content of a txt/csv file when running script

Putting all the Data in one Cell is indeed one way, using '!DATASOURCE' + 'FILEDELETE' + 'SAVEAS' (in this order...!) and there isn't hardly any Difficulty, so I don't really understand your "Problem is that when the time comes to save the data back to the same .csv file using a js variable there doesn't seen to be a way to do it.", unless you encountered this Bug/Limitation concerning 'FILEDELETE' with v9.0.3:
- FILEDELETE overwrite stopped (?) working

Variations include opening your File as a '.TXT' File in some second Tab in your Browser for "instant" Extraction, or as '.HTML' File saved from Excel/OO (from a '.CSV' File), or looping through your DataSource from your '.js' Script (doable with up to 50-100 Rows, after it becomes a bit too slow), or using an Online GoogleDoc File as your File..., but I guess I already mentioned all those Possibilities in the first Thread I referred you to...
Last edited by chivracq on Sun Nov 20, 2016 12:23 pm, edited 1 time in total.
- (F)CIM = (Full) Config Info Missing: iMacros + Browser + OS with all 3 Versions...
- I usually don't even read the Question if that (required) Info is not mentioned...
- Script & URL usually help a lot for a more "educated" Help...
chivracq
 
Posts: 6373
Joined: Sat Apr 13, 2013 6:07 am
Location: Amsterdam (NL)

Re: Saving to and importing strings from local text file

by microuser3 on Sun Nov 20, 2016 11:39 am

Thanks a lot for your reply, I appreciate. I am currently going through the links your gave me.

OK, I get the Scenario, I only notice that the "modify it" part from your first Description in "... I need to be able to save this data locally, and be able to fetch it, modify it, use includes() method on the file, append new data to it, etc...", which could be a tricky part has disappeared from your second Description where you basically only want to "add" (= append) some new Data.


I still need to do that :P I don't see it as a tricky part though, as long as I am able to read the data file and assign it to a js variable.

Putting all the Data in one Cell is indeed one way, using '!DATASOURCE' + 'FILEDELETE' + 'SAVEAS' (in this order...!) and there isn't hardly any Difficulty, so I don't really understand your "Problem is that when the time comes to save the data back to the same .csv file using a js variable there doesn't seen to be a way to do it.", unless you encountered this Bug/Limitation concerning 'FILEDELETE' with v9.0.3:


I read in the documentation that SAVEAS can only be used to save the last EXTRACT, which would make it hard to have it save a variable. Unless there is some way to trick it into making the last EXTRACT the variable you want to save..


EDIT:
I read the threads you linked, I don't think they quite fit what im trying to do. They are using .csv and appending new commas separated EXTRACT data to it whereas I need to save data from a js variable. I could possibly save all my data entries in individual .csv cells, but then I would have no way of knowing if the new entry I'm about to add already exists in that file(i don't want duplicates). And also another problem might be number of columns, my project will need about 200 "ID number" entries.

As for using a .txt file and opening it in a new tab, how could I then append data to it and save it?
Last edited by microuser3 on Mon Nov 21, 2016 9:50 am, edited 2 times in total.
microuser3
 
Posts: 11
Joined: Sat Nov 19, 2016 6:09 pm

Re: Saving to and importing strings from local text file

by microuser3 on Sun Nov 20, 2016 7:19 pm

Just an update. I have, I think found a workaround by using ADD !EXTRACT {{everything that was already in the file before}}
and then deleting the old file and then saving the new one from the extract variable. It looks like this:

Code: Select all
SET !EXTRACT NULL
TAG XPATH="{{somepath}}" EXTRACT=TXT
ADD !EXTRACT {{HISTORY}}
FILEDELETE NAME=Datasources\ID.csv
SAVEAS TYPE=EXTRACT FOLDER=C:\Users\...\iMacros\Datasources FILE=ID.csv


Unfortunately it seems I have hit a brick wall. It turns out that for some reason both !DATASOURCE and FILEDELETE will only work when a path is not specified, leading them to their default paths, which are respectively \Datasources and \Downloads. Even specifying their default paths doesnt work, example:
Code: Select all
SET !DATASOURCE C:\Users\...\iMacros\Datasources\ID.csv
'wont work

'but the folloing will, even though its the same path
SET !DATASOURCE ID.csv



Not sure if its a bug with my current version or whatever but sure is rly frustrating. I'd love someone to tell me I'm just doing it wrong but i have read the documentation and it seems I'm doing everything right :(



*for anyone interested here is the error I get when using FILEDELETE with a path, for example:
Code: Select all
'input code
FILEDELETE NAME=Datasources\somefile.csv

'error message
, line: 3 (Error code: -1001)
microuser3
 
Posts: 11
Joined: Sat Nov 19, 2016 6:09 pm

Re: Saving to and importing strings from local text file

by microuser3 on Mon Nov 21, 2016 1:34 pm

I managed to find a workaround for the bug, and also one for the SAVEAS limitations, I now have a functional database all in one string I can load/modify/save at will with js. I will post it here in a bit, for posterity.
microuser3
 
Posts: 11
Joined: Sat Nov 19, 2016 6:09 pm

Re: Saving to and importing strings from local text file

by chivracq on Mon Nov 21, 2016 11:46 pm

microuser3 wrote:Thanks a lot for your reply, I appreciate. I am currently going through the links your gave me.

OK, I get the Scenario, I only notice that the "modify it" part from your first Description in "... I need to be able to save this data locally, and be able to fetch it, modify it, use includes() method on the file, append new data to it, etc...", which could be a tricky part has disappeared from your second Description where you basically only want to "add" (= append) some new Data.


I still need to do that :P I don't see it as a tricky part though, as long as I am able to read the data file and assign it to a js variable.

Putting all the Data in one Cell is indeed one way, using '!DATASOURCE' + 'FILEDELETE' + 'SAVEAS' (in this order...!) and there isn't hardly any Difficulty, so I don't really understand your "Problem is that when the time comes to save the data back to the same .csv file using a js variable there doesn't seen to be a way to do it.", unless you encountered this Bug/Limitation concerning 'FILEDELETE' with v9.0.3:


I read in the documentation that SAVEAS can only be used to save the last EXTRACT, which would make it hard to have it save a variable. Unless there is some way to trick it into making the last EXTRACT the variable you want to save..


EDIT:
I read the threads you linked, I don't think they quite fit what im trying to do. They are using .csv and appending new commas separated EXTRACT data to it whereas I need to save data from a js variable. I could possibly save all my data entries in individual .csv cells, but then I would have no way of knowing if the new entry I'm about to add already exists in that file(i don't want duplicates). And also another problem might be number of columns, my project will need about 200 "ID number" entries.

As for using a .txt file and opening it in a new tab, how could I then append data to it and save it?

microuser3 wrote:Just an update. I have, I think found a workaround by using ADD !EXTRACT {{everything that was already in the file before}}
and then deleting the old file and then saving the new one from the extract variable. It looks like this:

Code: Select all
SET !EXTRACT NULL
TAG XPATH="{{somepath}}" EXTRACT=TXT
ADD !EXTRACT {{HISTORY}}
FILEDELETE NAME=Datasources\ID.csv
SAVEAS TYPE=EXTRACT FOLDER=C:\Users\...\iMacros\Datasources FILE=ID.csv


Unfortunately it seems I have hit a brick wall. It turns out that for some reason both !DATASOURCE and FILEDELETE will only work when a path is not specified, leading them to their default paths, which are respectively \Datasources and \Downloads. Even specifying their default paths doesnt work, example:
Code: Select all
SET !DATASOURCE C:\Users\...\iMacros\Datasources\ID.csv
'wont work

'but the folloing will, even though its the same path
SET !DATASOURCE ID.csv



Not sure if its a bug with my current version or whatever but sure is rly frustrating. I'd love someone to tell me I'm just doing it wrong but i have read the documentation and it seems I'm doing everything right :(

*for anyone interested here is the error I get when using FILEDELETE with a path, for example:
Code: Select all
'input code
FILEDELETE NAME=Datasources\somefile.csv

'error message
, line: 3 (Error code: -1001)

microuser3 wrote:I managed to find a workaround for the bug, and also one for the SAVEAS limitations, I now have a functional database all in one string I can load/modify/save at will with js. I will post it here in a bit, for posterity.

Hum, quoting all the Replies, I'll check the Content "tomorrow"..., a bit tired now, ah-ah...! But OP is learning/improving, good/good, ah-ah..!
(Interesting Thread, I'm following it...)
- (F)CIM = (Full) Config Info Missing: iMacros + Browser + OS with all 3 Versions...
- I usually don't even read the Question if that (required) Info is not mentioned...
- Script & URL usually help a lot for a more "educated" Help...
chivracq
 
Posts: 6373
Joined: Sat Apr 13, 2013 6:07 am
Location: Amsterdam (NL)

Re: Saving to and importing strings from local text file

by microuser3 on Sat Nov 26, 2016 2:22 pm

OK I will be quick and concise.

First the workaround for the bug is to simply go in Settings>Paths and set everything to iMacros\Datasources.

Here's how to setup your database. Basically you only use two columns, one for the last added values and one for all the rest.
So load both columns, something like that:
Code: Select all
var load;
load =  "CODE:";
load +=  "SET !EXTRACT null" + "\n";
load +=  "SET !DATASOURCE mydatabasefile.csv" + "\n";
load +=  "SET !EXTRACT {{!col1}}" + "\n";
load +=  "ADD !EXTRACT {{!col2}}" + "\n";
iimPlay(load);
var mydatabase=iimGetLastExtract();
mydatabase=mydatabase.replace(/\[EXTRACT\]/g,'');


This basically joins the two colums into one string, you can use some more JS to arrange it how you want.

when you want to add values, do something like this:
Code: Select all
SET !EXTRACT {{NEWDATA}}
SET !EXTRACTADD {{HISTORY}}
FILEDELETE NAME=mydatabasefile.csv
SAVEAS TYPE=EXTRACT FOLDER=C:\...\iMacros\Datasources FILE=mydatabasefile.csv


Run this with JS though, with something like this:
Code: Select all
iimSet("HISTORY", mydatabase);
iimSet("NEWDATA", NewDataVariable);
iimPlay("save.iim");


Your .csv file will always look something like this:
"NEWDATA NEWDATA NEWDATA ","OLDDATA OLDDATA OLDDATA HERPDERP WHATEVERYOUALREADYHADINHERE"

Questions, comments, suggestions, improvements?
microuser3
 
Posts: 11
Joined: Sat Nov 19, 2016 6:09 pm

Re: Saving to and importing strings from local text file

by chivracq on Sun Nov 27, 2016 9:38 am

microuser3 wrote:OK I will be quick and concise.

First the workaround for the bug is to simply go in Settings>Paths and set everything to iMacros\Datasources.

Here's how to setup your database. Basically you only use two columns, one for the last added values and one for all the rest.
So load both columns, something like that:
Code: Select all
var load;
load =  "CODE:";
load +=  "SET !EXTRACT null" + "\n";
load +=  "SET !DATASOURCE mydatabasefile.csv" + "\n";
load +=  "SET !EXTRACT {{!col1}}" + "\n";
load +=  "ADD !EXTRACT {{!col2}}" + "\n";
iimPlay(load);
var mydatabase=iimGetLastExtract();
mydatabase=mydatabase.replace(/\[EXTRACT\]/g,'');


This basically joins the two colums into one string, you can use some more JS to arrange it how you want.

when you want to add values, do something like this:
Code: Select all
SET !EXTRACT {{NEWDATA}}
SET !EXTRACTADD {{HISTORY}}
FILEDELETE NAME=mydatabasefile.csv
SAVEAS TYPE=EXTRACT FOLDER=C:\...\iMacros\Datasources FILE=mydatabasefile.csv


Run this with JS though, with something like this:
Code: Select all
iimSet("HISTORY", mydatabase);
iimSet("NEWDATA", NewDataVariable);
iimPlay("save.iim");


Your .csv file will always look something like this:
"NEWDATA NEWDATA NEWDATA ","OLDDATA OLDDATA OLDDATA HERPDERP WHATEVERYOUALREADYHADINHERE"

Questions, comments, suggestions, improvements?

Yep very good + very good, ah-ah...!, you are learning quick and you are quite Creative, which is the best "Quality" to do "interesting" things with iMacros... And Thanks for sharing your Solution, of course... :D

OK, my "Feedback", ah-ah...!
1- So you went for the way that I qualified as "shouldn't present any difficulties unless you hit the Bug/Limitation about 'FILEDELETE' in v9.0.3", which you hit indeed, ah-ah...! I am a bit surprised that you associated it as well with '!DATASOURCE' as I'm not aware that '!DATASOURCE' has the same Bug/Limitation in v9.0.3 as afaik, this had not been reported for '!DATASOURCE' which is much more used than 'FILEDELETE'...
Perfect that you found a Workaround, going back to v8.9.7 would solve it as well... (much more stable than v9.0.3).

2- I noticed you were using '!EXTRACTADD', which I am nearly surprised is still working in v9.0.3, as it was deprecated many years ago already now, about 6 or 8 years ago I would think... Even if iMacros is good at keeping later Releases backward compatible with deprecated Commands...
Hum, and 'iimGetLastExtract()' as well officially got replaced by 'iimGetExtract()' a few years ago, even if there are still more Code Examples on the Forum using the first one...

3- In the current Implementation of your Solution, a '.js' Script was not really required, you could have done it completely in pure '.iim' (by doing all the String Data Manipulation using 'EVAL()', but OK, if you later want to add some/more Conditional Logic to your Script, the "standard" way to do so is indeed to use a '.js' Script, so your Solution is already ready for this "next step"... 8)
- (F)CIM = (Full) Config Info Missing: iMacros + Browser + OS with all 3 Versions...
- I usually don't even read the Question if that (required) Info is not mentioned...
- Script & URL usually help a lot for a more "educated" Help...
chivracq
 
Posts: 6373
Joined: Sat Apr 13, 2013 6:07 am
Location: Amsterdam (NL)

Re: Saving to and importing strings from local text file

by microuser3 on Sun Nov 27, 2016 10:16 pm

I noticed you were using '!EXTRACTADD', which I am nearly surprised is still working in v9.0.3, as it was deprecated many years ago already now, about 6 or 8 years ago I would think..

What should I use instead? just found this googling.
microuser3
 
Posts: 11
Joined: Sat Nov 19, 2016 6:09 pm

Re: Saving to and importing strings from local text file

by chivracq on Mon Nov 28, 2016 2:24 am

microuser3 wrote:
I noticed you were using '!EXTRACTADD', which I am nearly surprised is still working in v9.0.3, as it was deprecated many years ago already now, about 6 or 8 years ago I would think..

What should I use instead? just found this googling.

Well, simply check that '!EXTRCTADD Command on the Wiki, ah-ah...!! (... even if it's not referenced directly anymore in the Command Reference...)
- (F)CIM = (Full) Config Info Missing: iMacros + Browser + OS with all 3 Versions...
- I usually don't even read the Question if that (required) Info is not mentioned...
- Script & URL usually help a lot for a more "educated" Help...
chivracq
 
Posts: 6373
Joined: Sat Apr 13, 2013 6:07 am
Location: Amsterdam (NL)


Return to iMacros for Firefox

Who is online

Users browsing this forum: Bing [Bot], Google [Bot] and 6 guests

cron
-->