Fiddler + imacros multi-threading

Support for iMacros. The iMacros software is the unique solution for automating every activity inside a web browser, for data extraction and web testing.
Forum rules
Before asking a question or reporting an issue:
1. Please review the list of FAQ's.
2. Use the search box (at the top of each forum page) to see if a similar problem or question has already been addressed.
3. Try searching the iMacros Wiki - it contains the complete iMacros reference as well as plenty of samples and tutorials.
4. We can respond much faster to your posts if you include the following information:CLICK HERE FOR IMPORTANT INFORMATION TO INCLUDE IN YOUR POST
Post Reply
cjseriy
Posts: 16
Joined: Wed Jul 04, 2018 12:53 pm

Fiddler + imacros multi-threading

Post by cjseriy » Wed Jul 04, 2018 6:52 pm

Hi!

I want to debug a site using imacros and Fiddler (I am open to another solutions that can log a traffic and autosave a log). Never done it before.

It is a 1 site that I will run using 5-10 simultaneous firefox profiles.
What I cannot understand is how will I differentiate which log correspond to which firefox profile since they run simultaneously :?:


VERSION BUILD=11.5.498.2403
Windows 10
iMacros Browser + Firefox iMacros 8.9.7
Firefox 49.0.2
Last edited by cjseriy on Thu Jul 05, 2018 2:36 pm, edited 2 times in total.
chivracq
Posts: 8193
Joined: Sat Apr 13, 2013 1:07 pm
Location: Amsterdam (NL)

Re: Fiddler + imacros multi-threading

Post by chivracq » Wed Jul 04, 2018 7:26 pm

cjseriy wrote:

Code: Select all

VERSION BUILD=11.5.498.2403
Windows 10
Imacros Browser + Firefox Imacros 8.9
Hi!

I want to debug a site using imacros and Fiddler (I am open to another solutions that can log a traffic and autosave a log). Never done it before.

It is a 1 site that I will run using 5-10 simultaneous firefox profiles.
What I cannot understand is how will I differentiate which log correspond to which firefox profile since they run simultaneously :?:
I will "wait" before answering this 2nd Thread of yours until you've "finished" the first one, with a Solution etc...

Hum, for this one (and all other Threads you might open), having your FCI in your Sig denotes from a good Intention but you still need to hard-code it in your OP as you only have one Dynamic Sig for the whole Forum and it can actually become "confusing" for other Users reading this Thread in a few weeks/months/years and you'll have probably updated your Config by that time to some later Versions...
(And it would be nice if you could spell "iMacros" correctly, ah-ah...! :wink: )

Your iMacros Version is incomplete as well btw, will be v8.9.7, I reckon...? (and not "v8.9" which doesn't exist...), and your FF Version is missing as well... :idea:

>>>

+ Parallel Thread on SOF:
- How to use Fiddler and imacros with multi-threading
(No Replies yet...)
(You missed a "+1" from me on SOF for not mentioning your FCI, ah-ah...! :wink: )
- (F)CIM = (Full) Config Info Missing: iMacros + Browser + OS with all 3 Versions...
- I usually don't even read the Question if that (required) Info is not mentioned...
- Script & URL usually help a lot for a more "educated" Help...
cjseriy
Posts: 16
Joined: Wed Jul 04, 2018 12:53 pm

Re: Fiddler + imacros multi-threading

Post by cjseriy » Thu Jul 05, 2018 10:33 am

Sorry, I have updated my FCI :roll:
chivracq
Posts: 8193
Joined: Sat Apr 13, 2013 1:07 pm
Location: Amsterdam (NL)

Re: Fiddler + imacros multi-threading

Post by chivracq » Thu Jul 05, 2018 2:34 pm

cjseriy wrote:Sorry, I have updated my FCI :roll:
OK, very good...!, hum, nearly...! Not updated in your Sig, and "iMacros" is still not spelt correctly..., the Capital is on the "M", not on the "i", you are going to use/need this Term quite often on the Forum..., so try to spell it correctly from the beginning...
+ FCI mentioned on SOF, very-very good, ah-ah...! :D (You'll get your "+1" on SOF once "iMacros" will have been spelt there correctly as well, ah-ah...!)

EDIT_1: Ah-ah...!, some Update in your Sig while I was typing my (long) Reply, "iMacros" still not correct, ah-ah...! YES...!, you can do it...!! :wink:
EDIT_2: OK, good enough now, and even on SOF at least in your FCI... (=> "+1" on SOF, ah-ah...! 8) ) :D

>>>

OK, Interesting Qt btw, first Thread of this kind on the Forum I think, I don't recall seeing a similar Thread before... 8)

But hum..., some "Interesting" Thread you might want to have a look at, not exactly-directly related, but contains a lot of Links to different Tools that could be useful maybe...:
- Ignore Untrusted Certificates

Then a "Remark" about your FCI:
iMacros for FF v8.9.7 is still the "Stable" (Full) Version on FF, perfect...!, and it works until FF56, I have it on FF v55.0.3 myself, for example.
From FF57, you'll need v10.0.2 (in Beta for the moment)... (And v9.0.3 also only works until FF56. (But v8.9.7 is "better"...!))
=> Your FF49 looks a bit "strange"/"old" or at least not really "representative" (anymore) for many/most Users if you want to do some Testing on some/your Site, even if it's "good" to test indeed with different Browsers and different Versions of all Browsers.

But be aware that if you update FF49 to FF55 or FF56, that from FF54, Mozilla/Firefox updated as well the FF Profiles to a different Format, and that once updated to FF54+, it's not possible to "downdate" to your FF49 Profile anymore, so make sure to make a Backup of your FF49 Profile (with FF completely closed/killed, or some Files remain in Use by FF and get silently skipped when copying your Profile(s) which then renders your Backup useless if you ever want to reuse it later...!) before updating, ah-ah...! :idea: :!:
It's actually even better to use separate Profiles for each FF Version you want to maintain in parallel, FF is a bit of a "pain in the ass" when maintaining several Versions of FF or you'll constantly run into "Firefox is already running" Hassle, or Shortcuts even launch the "wrong" Version, it's a complete Mess, even with FF Portable which has not been "designed" to be used with different/separate (multiple) Profiles (it can be done, but that's another pain in the ass!), ah-ah...!

Then OK, I've never used 'Fiddler', so I don't know how 'Fiddler' logs its Traffic... (Hum, maybe you could mention btw which Version of 'Fiddler' you are/will be using...), and I guess when talking about "Logs", you mean the Fiddler's Log(s), have a look at the Wiki for 'PROXY', maybe you can specify a different (Local) Port for each FF-Profile, which might generate separate Log Files from Fiddler...

From iMacros itself, you can do some Logging as well, using the Built-in 'PROFILER', and/or building your own "customized" Logging in your Script(s) with 'EXTRACT', 'STOPWATCH' and gathering all the Info/Data you'll want to save to your Log (=> '.TXT' or '.LOG' File, or even '.CSV') using the 'SAVEAS TYPE=EXTRACT' Mechanism.

For each of your [5-10] FF-Profiles, you'll need to (re)install their separate iMacros for FF Add-on, but I guess you'll want to use some "Central" Folder Tree Structure with the same 'Macros' + 'Downloads' etc Folders and not maintain 5 or 10 separate Folder Sets.

Depending on how you'll be starting those Profiles (manually) from some Windows Shortcut, or from some '.BAT' File, you could have each Profile first open some mini-Local (separate) '.TXT' File identifying each Profile (either from its Filename in the URL (or using any "Fake" URL instead of a Local File), or re-extracting its Content like a "Standard" Web-Page) to then reuse that Identifier for setting '!FOLDER_DOWNLOAD' dynamically from your Script (that will be used by the 'Profiler' and 'SAVEAS') or for the Filename for the 'SAVEAS'.

Hum, I see that '!FOLDER_STOPWATCH' in the Wiki is (now...?) available for all 4 Browsers, I thought iMacros for FF could only save to the Default 'Downloads' Folder, but the Wiki Page was very recently modified, maybe that's for v9.0.3 or v10.0.2 for FF, I'm not sure it will work for v8.9.7..., or I might be missing "stg" or I'm getting a bit "mixed up" :oops: , ah-ah...! But I don't "really" use all that Functionality / those Commands in my Scripts, or I only "played" a bit with it a long time ago or for specific Threads, so I don't remember/know for sure...

Or on the same "Principle", it's possible to identify/re-extract your FF-Profile directly from a Script, like demonstrated a mini-while ago in this recent Thread...:
- Any Imacros Command/Script To Retrieve Firefox Profile Name
=> From 'about:profiles' or even better from 'about:cache'..., funny "Trick" actually, ah-ah...! 8)
(Hum, "recent" Thread, I thought it was from just a few weeks ago maybe, but that Thread is already from 6 months ago, oh jeez...! :shock: )

>>>

OK, those are already a few "Thoughts"..., pity I don't know 'Fiddler', I might then have some "better" Ideas if I knew how it works and what it does exactly, ah-ah...! :oops:
(But you might convince me to give it a "Try", ah-ah...! I had actually already downloaded it a few months ago, but I never installed it / started to "play" with it, oops...!)
- (F)CIM = (Full) Config Info Missing: iMacros + Browser + OS with all 3 Versions...
- I usually don't even read the Question if that (required) Info is not mentioned...
- Script & URL usually help a lot for a more "educated" Help...
cjseriy
Posts: 16
Joined: Wed Jul 04, 2018 12:53 pm

Re: Fiddler + imacros multi-threading

Post by cjseriy » Thu Jul 05, 2018 3:53 pm

chivracq wrote: "iMacros" will have been spelt there correctly as well, ah-ah...!)
Yes, I have corrected it :D

The main reason why I need Fiddler is to log and debug network post/get requests (their sequence and behavior) in order to understand some discrepancies in data. In particular I use it for Google Analytics = Google Tag Manager with conjunction of their own debuggers.
The same network information can be found under Network tab in a Firebug pane as I understand.
chivracq wrote: Your FF49 looks a bit "strange"/"old" or at least not really "representative" (anymore)
I have chosen FF49 because of the recommendation from here https://imacros.net/imacros-firefox-com ... -56-newer/
For the most functionality and best stability, we recommend downgrading to iMacros for Firefox 8.9.7 and staying with Firefox 49.0.2.
Besides for current debugging representativeness will not be a bug issue I think.
chivracq wrote: But hum..., some "Interesting" Thread you might want to have a look at, not exactly-directly related, but contains a lot of Links to different Tools that could be useful maybe...:
- Ignore Untrusted Certificates
Thank you! :) I will have a look at it.
chivracq wrote: But be aware that if you update FF49 to FF55 or FF56, that from FF54, Mozilla/Firefox updated as well the FF Profiles to a different Format, and that once updated to FF54+, it's not possible to "downdate" to your FF49 Profile anymore
Thank you for caution! I have created different iMacros folders for each FF profile and disables updates for FF and iMacros (for each profile).
I have installed only 1 version of FF without portable version.
chivracq wrote: maybe you can specify a different (Local) Port for each FF-Profile, which might generate separate Log Files from Fiddler...
Yes, I am thinking about it, I don't have experience with Fiddler, the main problem is that Fiddler have only 1 local proxy (may be I am wrong).
chivracq wrote: From iMacros itself, you can do some Logging as well, using the Built-in 'PROFILER'
Thank you, didn't know about it, but unfortunately it does not include network logs.

chivracq wrote: Depending on how you'll be starting those Profiles (manually) from some Windows Shortcut, or from some '.BAT' File, you could have each Profile first open some mini-Local (separate) '.TXT' File identifying each Profile (either from its Filename in the URL (or using any "Fake" URL instead of a Local File), or re-extracting its Content like a "Standard" Web-Page) to then reuse that Identifier for setting '!FOLDER_DOWNLOAD' dynamically from your Script (that will be used by the 'Profiler' and 'SAVEAS') or for the Filename for the 'SAVEAS'.
I have changed paths in iMacros add-on settings for each profile so that they all have already different locations.
chivracq wrote: Or on the same "Principle", it's possible to identify/re-extract your FF-Profile directly from a Script, like demonstrated a mini-while ago in this recent Thread...:
- Any Imacros Command/Script To Retrieve Firefox Profile Name
=> From 'about:profiles' or even better from 'about:cache'..., funny "Trick" actually, ah-ah...! 8)
(Hum, "recent" Thread, I thought it was from just a few weeks ago maybe, but that Thread is already from 6 months ago, oh jeez...! :shock: )
Hum, it is interesting. If Fiddler could save FF-Profile for each request then I would be able to match this data with the extracted data saved by iMacros. I will dig deeper.

Once again, thank you for your help! :)
cjseriy
Posts: 16
Joined: Wed Jul 04, 2018 12:53 pm

Re: Fiddler + imacros multi-threading

Post by cjseriy » Thu Jul 05, 2018 6:36 pm

Your code worked for me:

Code: Select all

TAG POS=1 TYPE=H3 ATTR=TXT:This<SP>profile<SP>is<SP>currently<SP>used*
TAG POS=R1 TYPE=TH ATTR=TXT:Root*
SET !EXTRACT NULL
TAG POS=R1 TYPE=TD ATTR=* EXTRACT=TXT
SET Root_Prf EVAL("var s='{{!EXTRACT}}'; var x,y,z; x=s.split('Profiles\'); y=x[1].split(' '); z=y[0]; z")

TAG POS=R1 TYPE=TH ATTR=TXT:Local*
SET !EXTRACT NULL
TAG POS=R1 TYPE=TD ATTR=* EXTRACT=TXT
SET Local_Prf EVAL("var s='{{!EXTRACT}}'; var x,y,z; x=s.split('Profiles\'); y=x[1].split(' '); z=y[0]; z")

PROMPT Root_Profile:<SP>_{{Root_Prf}}_<BR>Local_Profile:<SP>_{{Local_Prf}}_
Not sure about this line for English FF locale This<SP>profile<SP>is<SP>currently<SP>used . My FF locale is not in English.

The second part of the solution is that Fiddler shows id of a process like firefox:21668 and each FF profile will have its own id. I hope that this ids' are permanent.
Then if I know which FF profile correspond to which process id in Fiddler I can match them.
chivracq
Posts: 8193
Joined: Sat Apr 13, 2013 1:07 pm
Location: Amsterdam (NL)

Re: Fiddler + imacros multi-threading

Post by chivracq » Thu Jul 05, 2018 10:24 pm

cjseriy wrote:Your code worked for me:

Code: Select all

TAG POS=1 TYPE=H3 ATTR=TXT:This<SP>profile<SP>is<SP>currently<SP>used*
TAG POS=R1 TYPE=TH ATTR=TXT:Root*
SET !EXTRACT NULL
TAG POS=R1 TYPE=TD ATTR=* EXTRACT=TXT
SET Root_Prf EVAL("var s='{{!EXTRACT}}'; var x,y,z; x=s.split('Profiles\'); y=x[1].split(' '); z=y[0]; z")

TAG POS=R1 TYPE=TH ATTR=TXT:Local*
SET !EXTRACT NULL
TAG POS=R1 TYPE=TD ATTR=* EXTRACT=TXT
SET Local_Prf EVAL("var s='{{!EXTRACT}}'; var x,y,z; x=s.split('Profiles\'); y=x[1].split(' '); z=y[0]; z")

PROMPT Root_Profile:<SP>_{{Root_Prf}}_<BR>Local_Profile:<SP>_{{Local_Prf}}_
Not sure about this line for English FF locale This<SP>profile<SP>is<SP>currently<SP>used . My FF locale is not in English.

The second part of the solution is that Fiddler shows id of a process like firefox:21668 and each FF profile will have its own id. I hope that this ids' are permanent.
Then if I know which FF profile correspond to which process id in Fiddler I can match them.
Yeah...!, my Code always works, ah-ah...!, especially if I mention "Tested in xxx"... :twisted:

Hum, "always works", I actually notice a mini-Mistake in both 'EVAL()' (I've just corrected in the original Thread), you can better add a ';' after the final Return 'z' at the end of each 'EVAL()', v8.9.7 doesn't really need it, but it's better to have it, I'm currently Beta-testing v10.0.2 for FF and 'EVAL()' is becoming a pain-in-the-ass, all those 'EVAL()' won't work anyway anymore, you need (in v10.x) to store all 'EXTRACT''s in some Temp Var, navigate to some (Fake) 'google.com' URL for example where you then can do all 'EVAL()' as 'EVAL()' is not accepted anymore while your Browser is (still) on some 'about:xxx' Page, pfff..., useless...!! :shock:

OK, and your Script was for 'about:profiles', always handy to keep the URL in your Script... :idea:

But good News already, I had/have a few Replies/Ideas for/from your previous Reply, haven't had the time to post again... :wink:

>>>

OK, I'll do it here "a bit quick and dirty"... :wink:

- FF49 with v8.9.7:
Yeah, was some "old" Advice from TechSup, I never really understood why they mentioned FF49, that Advice came pretty "late" actually, when many Users were having Pb's with FF Multi-Process, I was already a few FF Versions later (working OK), I guess they only tried one FF Version, was working, so they posted it as "THE Advice", tja... :shock:

Only Reason(s) for FF49 are if you want to stay below FF54 for the Profile Update that I mentioned already...
+ Multi-Login (http/https) changed (annoyingly...!) around FF50 (can be solved/"tuned", but I see it again in FF60/FF61 that I use for v10.0.2 Beta for FF)...
+ And the iMacros 'FILTER' Command got broken by FF51 or FF52...
=> But otherwise, you can better use FF v55.0.3 like me, I would think... (v8.9.7 is supposed to work until FF56, but I didn't test it myself...)

- Ouf-ouf, every Profile its own iMacros Folder Set...! :shock: => Maintenance Nightmare in my Opinion...! :cry:
I maintain 3 iMacros for FF different Versions (v8.8.2, for my "Prod" Env. with Pale Moon v26.3.3 // + v8.9.7 + FF v55.0.3 for the Forum and a few Scripts that can't run "completely" on PM because of Flash // + v10.0.2 Beta + FF60/FF61 for Beta-testing v10) and I very deliberately use the same Folder Structure for all 3, plus even for iMacros for CR v8.4.4 => now v10.0.2 + FIO, and iMacros for IE v12.0, => all 5 Env. using the same Folder Tree Structure and the same Macros... :idea:
(Only for v10.0.2 Beta for FF, I had to "modify" a few Macros so they could run in v10, because of many Commands not being supported anymore, but I gave them a different Name with a specific Suffix, but I wouldn't think of maintaining 5 Sets of Folders, or even only 3, as I very-very rarely use CR and IE, only for Forum Cases very specific to those 2 Browsers that I absolutely cannot test on PM (or FF55)...)

- Could run some apart 'Fiddler' Instance for each FF Profile, ah-ah...! Dunno, maybe an Idea... :twisted:

- Could use Cookies for setting/passing which Profile is being used... (Each FF Profile uses its own separate Cookies, even for the same Site...)

- From your previous Thread, you have a way to "inject" (=create/add/modify/delete) some Field/Attribute/Data into your '.html'/'.php' Page from your iMacros Script, Data that you can retrieve from your Page itself and pass to your URL's for 'get()'/'post()' or any URL's as a "Fake" Param like in "?Profile=Profile_01&pageSize=20&...xxx" etc that Fiddler will log...
- (F)CIM = (Full) Config Info Missing: iMacros + Browser + OS with all 3 Versions...
- I usually don't even read the Question if that (required) Info is not mentioned...
- Script & URL usually help a lot for a more "educated" Help...
cjseriy
Posts: 16
Joined: Wed Jul 04, 2018 12:53 pm

Re: Fiddler + imacros multi-threading

Post by cjseriy » Sat Jul 07, 2018 11:42 am

chivracq wrote: => But otherwise, you can better use FF v55.0.3 like me, I would think... (v8.9.7 is supposed to work until FF56, but I didn't test it myself...)

- Ouf-ouf, every Profile its own iMacros Folder Set...! :shock: => Maintenance Nightmare in my Opinion...! :cry:
I maintain 3 iMacros for FF different Versions (v8.8.2, for my "Prod" Env. with Pale Moon v26.3.3 // + v8.9.7 + FF v55.0.3 for the Forum and a few Scripts that can't run "completely" on PM because of Flash // + v10.0.2 Beta + FF60/FF61 for Beta-testing v10) and I very deliberately use the same Folder Structure for all 3, plus even for iMacros for CR v8.4.4 => now v10.0.2 + FIO, and iMacros for IE v12.0, => all 5 Env. using the same Folder Tree Structure and the same Macros... :idea:
Hum, thank you for the advice, so I will store iMacros(different for each profile) files in one shared folder but will need to create different folders for Download and Log. That is ok? :)
You are using v8.8.2 with Pale Moon v26.3.3 Does it have its limitations comparing to FF 49 beside fixes of future updates?
I have also instaled it now because of Firefox's Memory Leak.
chivracq wrote: - Could use Cookies for setting/passing which Profile is being used... (Each FF Profile uses its own separate Cookies, even for the same Site...)
Yes, I think it's a good idea, not sure how to do that.
chivracq wrote: - From your previous Thread, you have a way to "inject" (=create/add/modify/delete) some Field/Attribute/Data into your '.html'/'.php' Page from your iMacros Script, Data that you can retrieve from your Page itself and pass to your URL's for 'get()'/'post()' or any URL's as a "Fake" Param like in "?Profile=Profile_01&pageSize=20&...xxx" etc that Fiddler will log...
A bit difficult for me, do you mean I need to write a JS scrip that will retrieve FF profile and will pass it to URL?
Post Reply