!ClipBoard skeleton template [feedback request]

Support for iMacros. The iMacros software is the unique solution for automating every activity inside a web browser, for data extraction and web testing.
Forum rules
Before asking a question or reporting an issue:
1. Please review the list of FAQ's.
2. Use the search box (at the top of each forum page) to see if a similar problem or question has already been addressed.
3. Try searching the iMacros Wiki - it contains the complete iMacros reference as well as plenty of samples and tutorials.
4. We can respond much faster to your posts if you include the following information: CLICK HERE FOR IMPORTANT INFORMATION TO INCLUDE IN YOUR POST
Post Reply
User avatar
Edoctoor
Posts: 34
Joined: Sun Nov 29, 2015 7:53 pm

!ClipBoard skeleton template [feedback request]

Post by Edoctoor » Thu Dec 24, 2015 10:16 am

All I am requested to get great information from volunteers is to simply be clear about my current system; well I will be more than happy to state that at the top of all my posts:
--------------------------
iMacros Version 11.0.246.4051
Browser FireFox 42.0
OS Windows 7 Home Premium - Service Pack 1

--------------------------

While Slapping this together... a lot will change... when I have a working starting point; I will edit the first
four posts to be consistant; then I intend on creating a fifth post asking for your suggestions, opinions and improvements... of course if you see something that needs attention, but all means... kindly provide your knownledge.


OVERVIEW
Ok, As a general support and Discussion I would like to create a project that
basically works like the following and I am wondering if it can be done.
I have many reasons for using the !Clipboard.. but the main one is that writing to a file from several firefox profiles/macros instances at a time would result in a file lock.. and the !Clipboard option will not have that issue. Secondly, each of the three iMacros have to loops for different durations, and different amounts/independ times of each loop. Lastly they are way too big to put in one iMacro... hense... the three iMacros.

Note: "I have already setup and configured my FF profiles" using the following:
http://kb.mozillazine.org/Command_line_arguments

That said; this is what I intend to do.

It is just the BATCH and the !CLIPBOARD lesson that I am trying to learn here.

A BATCH FILE THAT DOES THE FOLLOWING:
=====================================


"THE ID COLLECTOR" - Macro (A) running on multi FireFox profiles
Three in this case: (Profile001,Profile002,Profile003)

=============================
(1) Scans a webpage and gets a ID number
(2) Has a hardcoded identification code used as an integrity checker
.... For example 'TRUE001','TRUE002','TRUE003' depending on which browser is sending the Clipboard data.
(3) Copies the Code, and ID to the clipboard
.... For example 'TRUE, 123456789'
(4) Scans !ClipBoard for successful pass.. then gets another ID
----------- END Macro (A) ------------

"THE ID PROCESSOR" - Macro (B) running on FF profile (D)
=============================
(1) Checks the hardcoded identification code used as an integrity checker
(2) Splits the !Clipboard and uses the digits after the comma
.... Examples

Code: Select all

SET TrimmedBefore EVAL("\"{{!EXTRACT}}\".replace(/.+?(?=,),/g, \"\");")
SET TrimmedAfter EVAL("\"{{trimmedValue}}\".replace(/,.*$/g, \"\");")
(3) does work using ID
(4) Writes results to a unusedaccounts.csv file
(5) Sends a successful code depending on which 'TRUE001','TRUE002','TRUE003' was processed
.... !ClipBoard will now store... 'Next001','Next002','Next003' to signal the matching browser to scan next ID.
----------- END Macro (A) ------------

"THE ID TERMINATOR" - Macro (C) running on FF profile (E)
=============================
(1) Reads the unusedaccounts.csv file
(2) does work using ID and lets the webmaster manually choose do delete the
.... unused accounts.
----------- END Macro (A) ------------

So easy to flowchart... but I bet I will run into some very frustrating issues when trying to get from start to finish of this project... join in if your interested.
Last edited by Edoctoor on Tue Dec 29, 2015 10:18 am, edited 12 times in total.
iMacros Version 11.0.246.4051
Browser FireFox 42.0
OS Windows 7 Home Premium - Service Pack 1

Before starting a topic please read the following; thank you!
http://forum.imacros.net/viewtopic.php?f=20&t=3331#p8799
User avatar
Edoctoor
Posts: 34
Joined: Sun Nov 29, 2015 7:53 pm

Re: !ClipBoard skeleton template

Post by Edoctoor » Thu Dec 24, 2015 10:19 am

BATCH file processed so far (rough draft tested and working; however, not finished)

Also No. of times it should loop i have made dynamic depending on the lines in my csv file. So it first counts no. of lines and then passes it to for loop %cnt%

Code: Select all

Rem -------------- draft master.bat file -------------- 

@echo off
@echo ========================================================================================
@echo Created by Edoctoor 2015
@echo 
@echo Shows how to use !Clipboard to pass several variables
@echo See [Solved] !ClipBoard skeleton template
@echo http://forum.imacros.net/viewtopic.php?f=2&t=25532&p=66924#p66924
@echo ========================================================================================

Rem   ========================================================================================
@echo Initialize variables
@echo ========================================================================================

doskey
Rem Delay in seconds Very Long
set /a DelayVL=40
Rem Delay in seconds Long
set /a DelayL=30
Rem Delay in seconds Short
set /a DelayS=5
Rem Variable to store the system path to the unusedaccounts.csv file
set file=E:%HOMEPATH%\Documents\iMacros\Macros\unusedaccounts.csv
Rem Each profile requires a "Profile Identification Code"
set pic=E:\Users\user\Documents\iMacros\Datasources\master.csv
Rem Encoding http://stackoverflow.com/questions/16598785/save-text-file-in-utf-8-encoding-using-cmd-exe
Rem UTF-8 ; "chcp 65001".... change back to the default encoding for command prompt "chcp 1252" when done.
chcp 65001

Rem   ========================================================================================
Rem   The following will count the number of lines in file and pass the count to the for loop
Rem   I have used %HOMEPATH% so my code should run on your system... however, my macros are on 
Rem   drive E:... so you may to have to edit the " E " that to " C " 
Rem   if that is where your iMacros are stored; example:
Rem   set file=C:\Users\account\Documents\iMacros\unusedaccounts.csv
Rem   ========================================================================================

@echo Step one start (Profile001,Profile002,Profile003) running macro-a.iim
@echo ========================================================================================
Rem   I have created many FF profiles all ending in 001, 002, 003, 004 etc..
Rem   Start three profiles (Profile001,Profile002,Profile003) running macro-a.iim
Rem   Interesting Note: When the Firefox browser first opens; the iMacros plugin takes a split second to load
Rem   so I delayed my iMacros macro from loading using the Firefox " -new-tab " option which allows 
Rem   the iMacros to load..(yes, my idea, LOL)

echo Profile001 > %pic%
timeout /t 1
start firefox.exe -no-remote -P "Profile001" -new-tab imacros://run/?m=macro-a.iim -tray -foreground
Rem   Please wait 30 seconds... then start new profile
timeout /t %DelayL%

echo Profile002 > %pic%
start firefox.exe -no-remote -P "Profile002" -new-tab imacros://run/?m=macro-a.iim -tray -foreground
Rem   Please wait 30 seconds... then start new profile
timeout /t %DelayL%

echo Profile003 > %pic%
start firefox.exe -no-remote -P "Profile003" -new-tab imacros://run/?m=macro-a.iim -tray -foreground
Rem   Please wait 30 seconds... then start new profile
timeout /t %DelayL%

@echo This for loop counts the lines in the unusedaccounts.csv file
@echo ========================================================================================
set /a cnt=0
for /f %%a in ('type "%file%"^|find "" /v /c') do set /a cnt=%%a
echo %file% has %cnt% lines
timeout /t %DelayS%

@echo This for loop processes the lines in the unusedaccounts.csv file
@echo ========================================================================================
for /L %%A in (1,1,%cnt%) do (
	@echo call the %cnt% macro script

        Rem Problem: if profile??? is used then it doesn't work... due to Firefox profile lock
        Rem by design... not a bug... but I haven't found a solution to bypass this profile lock
	Rem as a result, the following will not work:
        Rem start  /B "" firefox.exe -no-remote -P "Profile004" -new-tab imacros://run/?m=macro-b.iim

	Rem The good news is that we only require one profile to work within the loop and fortunately
	Rem the FireFox file lock isn't an issue with the "DEFAULT" Firefox profile so the following works fine.
	start /B "" "C:\Program Files (x86)\Mozilla Firefox\firefox.exe" imacros://run/?m="macro-b.iim"
	timeout /t %DelayVL%

	Rem delete 1st line of csv file at each loop
	more +1 "%file%" > "%file%.new"
	move /y "%file%.new" "%file%" > nul
)
Rem to re-set default encoding for command prompt back to "chcp 1252"
chcp 1252

Rem ------------- end of master.bat file -------------
 
Currently iMacros can read data sources files only in CSV format.
Each Profile loaded must have its own "Profile Identification Code"
E:\%HOMEPATH%\Documents\iMacros\Datasources\master.csv
This is just a example of the contents of master.csv which the batch file creates.

Code: Select all

Profile001

Manually created unusedaccounts.csv the file and saved in notepad using UTF-8 encoding; please note, I have heard that saving it with an excel program can cause some random issues.

If your seeking for more information/options related to delimiters, please read the following:
http://wiki.imacros.net/!DATASOURCE#Notes

Code: Select all

TRUE, 123456789
TRUE, 223456789
TRUE, 323456789
TRUE, 423456789
TRUE, 523456789
TRUE, 623456789
TRUE, 723456789
TRUE, 823456789
Last edited by Edoctoor on Sat Dec 26, 2015 6:41 am, edited 19 times in total.
iMacros Version 11.0.246.4051
Browser FireFox 42.0
OS Windows 7 Home Premium - Service Pack 1

Before starting a topic please read the following; thank you!
http://forum.imacros.net/viewtopic.php?f=20&t=3331#p8799
User avatar
Edoctoor
Posts: 34
Joined: Sun Nov 29, 2015 7:53 pm

Re: !ClipBoard skeleton template

Post by Edoctoor » Thu Dec 24, 2015 10:20 am

Stub for Macro (A) (Rought draft Not even close to the finished product)

NAMED "THE ID COLLECTOR"

Code: Select all

' ===============================================================
' ===============================================================
' ===============================================================

TAB T=1

' Rought Goal of macro-a.iim
' ========================================================================
'Macro (A) running on FF profile (A)
'Macro (A) running on FF profile (B)
'Macro (A) running on FF profile (C)
'=============================
'(1) Scans a webpage and gets a ID number
'(2) Has a hardcoded identification code used as an integrity checker
'.... For example 'TRUE001','TRUE002','TRUE003' depending on which browser is sending the Clipboard data.
'(3) Copies the Code, and ID to the clipboard
'.... For example 'TRUE, 123456789'
'(4) Scans !ClipBoard for successful pass.. then gets another ID
'----------- END Macro (A) ------------



' ========================================================================
' Initialize standard macro starting configurations
' ========================================================================
' Make tab 1 active
TAB T=1
' Macros are more stable if all other tabs are closed
TAB CLOSEALLOTHERS
' Don't stop if a button is missing or if page hasn't loaded
SET !ERRORIGNORE 		NO
' If button is missing don't wait a minute for the button to appear
SET !TIMEOUT_STEP 		0
' Don't show popups 
SET !EXTRACT_TEST_POPUP 	YES
' No speed limits enabled
SET !REPLAYSPEED FAST
' Clear contents stored in !EXTRACT in case of loop
' Setting !WAITPAGECOMPLETE to YES or NO (This will create an error in Firefox, disabled)
' SET !WAITPAGECOMPLETE YES
SET !FOLDER_DATASOURCE E:\Users\user\Documents\iMacros\Datasources\
SET !DATASOURCE E:\Users\user\Documents\iMacros\Datasources\master.csv

SET !EXTRACT 			NULL
' Profile Identification Code
SET !VAR1 			NULL
SET !VAR2 			NULL
SET !VAR3 			NULL
SET !VAR4 			NULL
SET !VAR5 			NULL
SET !VAR6 			NULL
SET !VAR7 			NULL
SET !VAR8 			NULL
SET !VAR9 			NULL
SET !CLIPBOARD 			Ready001

' User defined Variables
' ----------------------

' Set VAR1 with "Profile Identification Code" imported from the {{!DATASOURCE}}
SET !LOOP 1
SET !VAR1 {{!COL1}}
' To store ID
SET IDFound  NULL
SET RWeReady Ready001

' Start of Macro
' --------------

' :LOOP

' do stuff
' Flowchart
' Find ID, waits until change in Clipboard = "Ready001" then continue
' pass profile code, and ID to clipboard
' and scrap and prepare to send next ID
'
'Copy data from the clipboard
'SET RWeReady {{!CLIPBOARD}}
'
'Copy data to the clipboard:
'SET !CLIPBOARD {{!EXTRACT}}

' Scrap something... get ID
' -------------------------
' Bypass web scrapping "for now" and create a number from (100000000 - 999999999)
SET IDFound EVAL("var randomNumber=Math.floor(Math.random()*899999999 + 100000000); randomNumber;")	

' SET !VAR2 EVAL("var x='{{!EXTRACT}}'; var y = x.split(' '); var j = x.length; var i = 0; var s = 'id='; for(i;i<j;i++){if(y[i].indexOf(s) != -1){var myId = y[i].replace('id=', ''); myId = myId.replace(/\"/g, \"\"); i=j; }} myId;")
' SET !VAR2 EVAL("var i = 0; var s = 'id='; for(i;i<j;i++){if( '{{!CLIPBOARD}}' == '{{RWeReady}}' ){ i=j; } myId;")
' SET !VAR5 EVAL(" var randomNumber=Math.floor(Math.random()*10 + 1); randomNumber;")
' not this either... ummm var t = new Date().getTime(); while (new Date().getTime() < t + millisecondsToLockupBrowser); 

PROMPT {{!CLIPBOARD}}
'SET !VAR2 EVAL("while ( '{{!CLIPBOARD}}' != '{{RWeReady}}' ) ping google.com;;")
SET !VAR2 EVAL("var t = {{!CLIPBOARD}}   ;while ( 't' != '{{RWeReady}}' );")


PROMPT {{!CLIPBOARD}}

SET !CLIPBOARD {{IDFound}}
'copy Profilecode Identification Code, and next ID to the !Clipboard
' Scrap something... get ID
' -------------------------
' Bypass web scrapping "for now" and create a number from (100000000 - 999999999)
SET IDFound EVAL("var randomNumber=Math.floor(Math.random()*899999999 + 100000000); randomNumber;")	

'GOTO LOOP


' Ok, what are my options... ????
' I am trying to test the contents of {{!CLIPBOARD}} every second and exit (wait loop) until master macro stores "Ready001" in the
' {{!CLIPBOARD}}

' any suggestion would be very welcomed..





' Debug... list variables here to verify data stored in memory
' ------------------------------------------------------------
'PROMPT {{!DATASOURCE}}
'PROMPT {{!FOLDER_DATASOURCE}}
'PROMPT {{IDFound}}
' Profile Identification Code
'PROMPT {{!VAR1}}

' ===============================================================
' ===============================================================
' ===============================================================
Last edited by Edoctoor on Wed Jan 06, 2016 8:58 pm, edited 12 times in total.
iMacros Version 11.0.246.4051
Browser FireFox 42.0
OS Windows 7 Home Premium - Service Pack 1

Before starting a topic please read the following; thank you!
http://forum.imacros.net/viewtopic.php?f=20&t=3331#p8799
User avatar
Edoctoor
Posts: 34
Joined: Sun Nov 29, 2015 7:53 pm

Re: !ClipBoard skeleton template

Post by Edoctoor » Thu Dec 24, 2015 10:22 am

Stub for Macro (B) (Rought draft Not even close to the finished product)

NAMED "THE ID PROCESSOR"

This has to use the "DeFAULT" Firefox... because only the DeFAULT firefox profile doesn't have folder/file lock and this means that I can call the macro many times.. and each time it will load a tab... with an instant
of the macro... and then close the tab when done... go to next tab... well that is sorta what I am imaging.

I know how to get all this to work with multipal files... etc... but hopeful.. this overview is enough information that I can get some feedback with regards to if this can be done using the !CLIPBOARD
Last edited by Edoctoor on Sun Dec 27, 2015 5:41 pm, edited 6 times in total.
iMacros Version 11.0.246.4051
Browser FireFox 42.0
OS Windows 7 Home Premium - Service Pack 1

Before starting a topic please read the following; thank you!
http://forum.imacros.net/viewtopic.php?f=20&t=3331#p8799
User avatar
Edoctoor
Posts: 34
Joined: Sun Nov 29, 2015 7:53 pm

Re: !ClipBoard skeleton template

Post by Edoctoor » Thu Dec 24, 2015 10:23 am

Stub for Macro (C) (Rought draft Not even close to the finished product)

NAMED "THE ID TERMINATOR"

This one isn't necessary... however... during the testing design and testing stage... the only actions that really get done is here.. and it is done manually... to see if the macros are 100% trustworthy...

This is sorta like running the msdos command dir *.txt... before your run the command delete *.txt
because undelete is a real pain.

batch file.. lostfolders.bat (just interesting code... has nothing to do with the current project)

Code: Select all

@echo off
doskey
dir
set dircmd=0
pause
echo now try the dir command
dir
echo... where did your files and folders go....
echo... no worries they are still there.. we just disabled the dir command
Pause
set dircmd=1
dir
Echo this is just a useless trick I played on my computer support when 
Echo computers screens were black with green text, LOL
Last edited by Edoctoor on Sun Dec 27, 2015 5:54 pm, edited 3 times in total.
iMacros Version 11.0.246.4051
Browser FireFox 42.0
OS Windows 7 Home Premium - Service Pack 1

Before starting a topic please read the following; thank you!
http://forum.imacros.net/viewtopic.php?f=20&t=3331#p8799
User avatar
Edoctoor
Posts: 34
Joined: Sun Nov 29, 2015 7:53 pm

Re: !ClipBoard skeleton template

Post by Edoctoor » Sat Dec 26, 2015 12:21 pm

First question:::: This relates to the Macro-A above

Stub for Macro (A) (Rought draft Not even close to the finished product)

NAMED "THE ID COLLECTOR"
Edoctoor wrote:

Code: Select all

' Locks up browser
'SET !VAR2 EVAL("var r = '{{RWeReady}}'; var j = 10; var i = 0; for(i;i<j;i++){var s = {{!CLIPBOARD}};if ( s != r) {i++;}; i=j; }; r;")
How can I (check/wait) for the value that stored in {{!CLIPBOARD}} every ten seconds... and then continue when the value in {{!CLIPBOARD}} has changed and a match has occured?

If I loop it from the batch file... I'll have to OPEN and CLOSE the browser every iteration....
and javascript can not be made to wait.. as it locks up the browser... so bad programing.. and not useful.

What methods or options do I have to research to get this task rolling?

My guess is macro-a.js calling the macro... and not imacros invoking java... am I close to a solution?

Is there a solution? can it be done? are there other options? Lastly, I am really interested in examples of the following method:
chivracq wrote: Pfff..., instead of using a Temp File, it's possible to use the Clipboard ('!CLIPBOARD'), but I always find that dangerous especially if you keep working on the Computer while your Script is working, or it is possible to write and read some Variables using added (fake) Parameters to the URL with 'URL GOTO' and '!URLCURRENT', as I mentioned earlier if the Web-Server doesn't redirect you automatically to some generic 404 URL...
Lastly, I am not spamming... or doing anything wrong... I just don't want anyone to know what I am doing as my work is useless if two people are using my code.
Last edited by Edoctoor on Mon Dec 28, 2015 4:55 pm, edited 1 time in total.
iMacros Version 11.0.246.4051
Browser FireFox 42.0
OS Windows 7 Home Premium - Service Pack 1

Before starting a topic please read the following; thank you!
http://forum.imacros.net/viewtopic.php?f=20&t=3331#p8799
User avatar
Edoctoor
Posts: 34
Joined: Sun Nov 29, 2015 7:53 pm

Re: !ClipBoard skeleton template

Post by Edoctoor » Mon Dec 28, 2015 4:51 pm

I have giving a 24 hours of thought into my thread with 5 post explaining my problem... I have given my version numbers... there has been over 60 views... Is there a reason that no one is giving me a hint.. or pointing me in the right direction? Or is this simply the wrong use of !Clipboard... knock knock... is anyone home?

I thought that If I gave a clear overview... that response time would improve... what am I doing that is so wrong?

All prior... is overview.. this is the ONLY question:


How can I (check/wait) for the value that stored in {{!CLIPBOARD}} every ten seconds... and then continue when the value in {{!CLIPBOARD}} has changed and a match has occured?
iMacros Version 11.0.246.4051
Browser FireFox 42.0
OS Windows 7 Home Premium - Service Pack 1

Before starting a topic please read the following; thank you!
http://forum.imacros.net/viewtopic.php?f=20&t=3331#p8799
User avatar
Edoctoor
Posts: 34
Joined: Sun Nov 29, 2015 7:53 pm

Re: !ClipBoard skeleton template [feedback request]

Post by Edoctoor » Wed Jan 06, 2016 8:59 pm

Nothing?
iMacros Version 11.0.246.4051
Browser FireFox 42.0
OS Windows 7 Home Premium - Service Pack 1

Before starting a topic please read the following; thank you!
http://forum.imacros.net/viewtopic.php?f=20&t=3331#p8799
chivracq
Posts: 8719
Joined: Sat Apr 13, 2013 1:07 pm
Location: Amsterdam (NL)

Re: !ClipBoard skeleton template

Post by chivracq » Wed Jan 06, 2016 11:59 pm

Edoctoor wrote:I have giving a 24 hours of thought into my thread with 5 post explaining my problem... I have given my version numbers... there has been over 60 views... Is there a reason that no one is giving me a hint.. or pointing me in the right direction? Or is this simply the wrong use of !Clipboard... knock knock... is anyone home?

I thought that If I gave a clear overview... that response time would improve... what am I doing that is so wrong?

All prior... is overview.. this is the ONLY question:


How can I (check/wait) for the value that stored in {{!CLIPBOARD}} every ten seconds... and then continue when the value in {{!CLIPBOARD}} has changed and a match has occured?
Oh sorry, I had had a look at your Thread about 1 week ago when you had started it but I was waiting for you to complete it a bit (Stubs A-B-C were still empty) and I had not noticed you had worked again on it (that's the "danger" of reserving and editing Posts, your Thread doesn't get bumped up in the Sub-Forum View...).

OK, lots of reading, interesting though... About your Question about waiting for 10 sec and then continue, I guess you mean some "Conditional WAIT", search my Posts on those Terms, I've already answered several Threads about that, with Examples. I use it myself, in pure '.iim'.

For checking the Content of the Clipboard for a Change, that's not complicated, you just store it some Temp Var and use 'EVAL()' to compare.

BUT...!, if "continue" means looping the Macro again, you will lose the Content of your Vars, because they all get reset between 2 Loops, '!EXTRACT' as well btw..., meaning that between 2 Loops, you won't know if the Current Value of '!CLIPBOARD' is still the old one or if it is a new one..., and you will still need to write to a Temp File to make sure you don't lose track. You should have the ID Collector Macro write all ID's to a .CSV File anyway, and the ID Processor as well...
When writing stg to the Clipboard, you could add the Timestamp and actually before (over)writing stg to it, first read its Content and keep the last Content together with the new one (ID + Timestamp), and/or still write it as well to a Temp File which would allow an extra Level of Check, that both the Content of the Clipboard and of the Temp File must be identical. I do use such a Mechanism myself (without the Clipboard) for a Macro that runs/loops between 1-15 times (it is able to decide by itself how many times it needs to run at the first Loop) and that keeps some aggregated Data that will be used only at the last Loop for some Report, in case anything goes wrong (bad Connection etc...) in any Loop.

I'm still not convinced that using the Clipboard is better than using Temp Files, I wanted to run a Test with one of my Macros that loops about 1000 times in 10-12 min to save some Data to a .CSV File for those 1000 Items, that's about 100 times the same 'SAVEAS' per Minute, that's nearly 2 'SAVEAS' per Second, by running it from 3 Profiles at the same time to see if any Data was missing (= not being saved 3 times).
- (F)CI(M) = (Full) Config Info (Missing): iMacros + Browser + OS (+ all 3 Versions + 'Free'/'PE').
- I don't even read the Qt if that (required) Info is not mentioned...!
- Script & URL help a lot for more "educated" Help...
Post Reply