How to increase TAG POS by 4 automatically?

Support for iMacros. The iMacros software is the unique solution for automating every activity inside a web browser, for data extraction and web testing.
Forum rules
Before asking a question or reporting an issue:
1. Please review the list of FAQ's.
2. Use the search box (at the top of each forum page) to see if a similar problem or question has already been addressed.
3. Try searching the iMacros Wiki - it contains the complete iMacros reference as well as plenty of samples and tutorials.
4. We can respond much faster to your posts if you include the following information:CLICK HERE FOR IMPORTANT INFORMATION TO INCLUDE IN YOUR POST
chivracq
Posts: 8198
Joined: Sat Apr 13, 2013 1:07 pm
Location: Amsterdam (NL)

Re: How to increase TAG POS by 4 automatically?

Post by chivracq » Sun May 27, 2018 10:35 am

PBNSurprise wrote:hey so thanks for the response, the script works nicely to scrape facebook users that like a page. i've tried it on a few pages and it continues to work. this is the code that i ended up using:

Code: Select all

VERSION BUILD=8820413 RECORDER=FX
SET !ERRORIGNORE YES
SET !EXTRACT_TEST_POPUP NO
TAB T=1

SET Like_Search "Fazio's Cat Jewelry"
SET POS_Pix {{!LOOP}}

'TAG POS={{!LOOP}} TYPE=DIV ATTR=TXT:Likes<SP>{{Like_Search}}*

TAG POS=1 TYPE=DIV ATTR=TXT:Add<SP>FriendFriend<SP>request<SP>sentMore<SP>*
TAG POS=R-1 TYPE=* ATTR=* EXTRACT=TXT
TAG POS=R{{POS_Pix}} TYPE=IMG ATTR=SRC:https://*.fbcdn.net/*_n.jpg* EXTRACT=TXT
SET !EXTRACT NULL
TAG POS=R1 TYPE=A ATTR=HREF:*?ref=br_rs EXTRACT=TXT
SAVEAS TYPE=EXTRACT FOLDER=* FILE=file.csv
WAIT SECONDS=1
a few things i've noticed:

it turns out that it doesn't matter what you enter for SET Like_Search. I've tried "Fazio's Cat Jewelry" but scraped different pages and seems to still work.

i added wait 1 second at the end because once the code gets to the end of the page there needs to be some time to load more results, the page is infinite scroll sadly.
Yep, the 'WAIT=1' is wise indeed because of the Infinite Scrolling that I don't like either, if anything "goes wrong" at some point and the Page stops loading new Content, which happens regularly with FB, then you have to start all over again from the beginning...

Well, the 'Like_Search' Var containing your "Fazio's Cat Jewelry" is actually not used, I left it in the Script "to give you some Ideas", I said...

I actually had started with the Profile Pix from each User/Result to use directly on the "raw" '!LOOP', but it was not starting at 'POS=1' for the 1st User as the Script was catching as well your own Profile Pix, + recent/current Chats with Pix were interfering as well and shifting the '!LOOP', so I tried using your Search Terms as all Results contain a "Likes Fazio's Cat Jewelry + xxx", but that Approach was hitting the Modulo/Increment +4 Difficulty that I was actually trying to avoid, so I dropped that Approach and went back to the Profile Pix Approach and added an extra Anchor as the Containing 'DIV' to make sure the Script would only look in the Results and wouldn't be "disturbed" by some other Content on the Page...
PBNSurprise wrote:one thing i've noticed is that once i've scrapped around 500 users, the code tries to locate the initial TAG POS at the top of the screen. meaning, for some reason the code goes back to user number 1 and tries to scrape from there. not sure why this happens. it turns out that TAG POS here:

Code: Select all

TAG POS=1 TYPE=DIV ATTR=TXT:Add<SP>FriendFriend<SP>request<SP>sentMore<SP>* 
needs to be changed after scraping around 500 users.

i struggle to determine what exactly is the new TAG POS. i know the last user that was scraped because i can check the file. so what i did was test various TAG POS until i landed where i needed to be. it was like trial and error. what i arrived at was TAG POS=3660. this lands me on the user where the code stopped scraping. so i stop the macro and input this new tag and now it's:

Code: Select all

TAG POS=3660 TYPE=DIV ATTR=TXT:Add<SP>FriendFriend<SP>request<SP>sentMore<SP>*. 
i don't reload the page so i can still see all the users and where i was just scraping. so then i start the macro again on a 100,000 times loop and boom, it continues scraping where it left off

i'm not sure if there's a way to implement this change automatically. but it's not a big deal if there isn't. can you tell me how to find the TAG POS of an item on the webpage? If i knew the exact TAG POS was 3660, i could just make these quick changes as needed and fire the macro up again without having to trial and error

other than this i think the script is running fine, i plan to scrape 100k users

once again thanks for the support
Hum, about 'POS=3660' on the 'DIV' to run the Script again if it stopped for any reason, it is a bit "normal"..., the 'POS=1' corresponds to the main/outer Containing 'DIV' containing all Results on the Page, and from 'POS=2', they correspond to each Result, with the "Difficulty" that each Result has 4 inner 'DIV''s I think that match the Criteria, so you would need to use 'POS=2' for the 1st Result, then 'POS=5' for the 2nd Result, etc...

=> If I'm correct about the x4 for each Result, your 'POS=3660' would mean that the Script had probably stopped at around '!LOOP'=915 (and not 500), and you could keep running the same Script again by just adding a "SET !LOOP 916" at the beginning of the Script to make it skip the first 915 Loops that already went fine.
An "easy" Implementation for this Approach is simply to save the current '!LOOP' in your 'SAVEAS' like in:

Code: Select all

VERSION BUILD=8820413 RECORDER=FX
SET !ERRORIGNORE YES
SET !EXTRACT_TEST_POPUP NO
TAB T=1

SET !LOOP 1
SET POS_Pix {{!LOOP}}

TAG POS=1 TYPE=DIV ATTR=TXT:Add<SP>FriendFriend<SP>request<SP>sentMore<SP>*
TAG POS=R-1 TYPE=* ATTR=* EXTRACT=TXT
TAG POS=R{{POS_Pix}} TYPE=IMG ATTR=SRC:https://*.fbcdn.net/*_n.jpg* EXTRACT=TXT
'>
SET !EXTRACT {{!LOOP}}
TAG POS=R1 TYPE=A ATTR=HREF:*?ref=br_rs EXTRACT=TXT
SAVEAS TYPE=EXTRACT FOLDER=* FILE=file.csv
WAIT SECONDS=1
=> When your Script stops, you check your '.CSV' to see which was the last Value for '!LOOP' that got correctly saved (say "915"), and you simply change the 'SET !LOOP 1' to 'SET !LOOP 916' and your Script should then go on from 916...

If you prefer to use the 'POS=3660' like you do, saving the '!LOOP' in the '.CSV' is still a good idea as I suspect there is a direct Correlation between '!LOOP=915' and 'POS=3660' (then you don't touch the '!SET LOOP 1') but your Results will be saved in the '.CSV' in "bunches" and the Counting on '!LOOP' will start again each time at "1".

Some other Approach would be to hard-code the Name of the first User/Result you want to start the Scrapping from to use this specific User as the first Anchor instead of the main Containing 'DIV' like I did or the inner 'DIV' like you do...
And that Approach would actually be more "reliable" I think as it doesn't "depend" on the Presence of an 'Add Friend' Button for the first User and it would still work if the first User doesn't have the 'Add Friend' Button or only has a 'Follow' Button... But you then each time need to edit the Script to hard-code the exact Name of that first Result, making sure you don't make any Typo and don't make any "Mistake" with Spaces (='<SP>')...
- (F)CIM = (Full) Config Info Missing: iMacros + Browser + OS with all 3 Versions...
- I usually don't even read the Question if that (required) Info is not mentioned...
- Script & URL usually help a lot for a more "educated" Help...
PBNSurprise
Posts: 15
Joined: Sun Jun 19, 2016 12:10 am

Re: How to increase TAG POS by 4 automatically?

Post by PBNSurprise » Sun May 27, 2018 9:54 pm

hey thanks for the reply, i will go ahead and test that. can you please let me know how the relative positioning works for tag POS? this is the code that i am working with now:

Code: Select all

VERSION BUILD=8820413 RECORDER=FX
SET !ERRORIGNORE YES
SET !EXTRACT_TEST_POPUP NO
TAB T=1

SET Like_Search "Fazio's Cat Jewelry"
SET POS_Pix {{!LOOP}}

'TAG POS={{!LOOP}} TYPE=DIV ATTR=TXT:Likes<SP>{{Like_Search}}*

TAG POS=1 TYPE=DIV ATTR=TXT:Add<SP>FriendFriend<SP>request<SP>sentMore<SP>*
TAG POS=R-1 TYPE=* ATTR=* EXTRACT=TXT
TAG POS=R-4 TYPE=* ATTR=* EXTRACT=TXT
TAG POS=R{{POS_Pix}} TYPE=IMG ATTR=SRC:https://*.fbcdn.net/*_n.jpg* EXTRACT=TXT
SET !EXTRACT NULL
TAG POS=R1 TYPE=A ATTR=HREF:*?ref=br_rs EXTRACT=TXT
TAG POS=R4 TYPE=A ATTR=HREF:*/pages/*?ref=br_rs EXTRACT=TXT
SAVEAS TYPE=EXTRACT FOLDER=* FILE=file.csv
WAIT SECONDS=1
it successfully scrapes two columns of data, however the second column is jumbled up with city,state and other data that i don't need. this is probably because my TAG POS=R is set incorrectly. i tried using position 4 and that works sometimes, but it seems the relative position of city,state for each profile changes. is this difficult to get around or is this an easy fix? sorry but i don't understand imacros quite well, i do believe that the issue lies within the TAG POS=R though, correctly identifying the correct relative position for city,state and scraping the data so that it matches the name. i'm going to continue to fiddle with this to see if it's something easy to solve but last resort i will may just go to upwork and hire a professional. if this is an easy fix please let me know so i can implement it right away and begin scraping, thanks!

EDIT: i think a better solution would be for me to figure out how to use tag relative position to target the last line of text for each profile
Image
that last line is where the city,state info will always be if it's available. will try to see how i can target that
PBNSurprise
Posts: 15
Joined: Sun Jun 19, 2016 12:10 am

Re: How to increase TAG POS by 4 automatically?

Post by PBNSurprise » Sun May 27, 2018 11:00 pm

well i fiddled with it for a bit and this is what i came up with:

Code: Select all

VERSION BUILD=8820413 RECORDER=FX
SET !ERRORIGNORE YES
SET !EXTRACT_TEST_POPUP NO
TAB T=1

SET !LOOP 1
SET POS_Pix {{!LOOP}}

'TAG POS={{!LOOP}} TYPE=DIV ATTR=TXT:Likes<SP>{{Like_Search}}*

TAG POS=1 TYPE=DIV ATTR=TXT:Add<SP>FriendFriend<SP>request<SP>sentMore<SP>*
TAG POS=R-1 TYPE=* ATTR=* EXTRACT=TXT
TAG POS=R-5 TYPE=* ATTR=* EXTRACT=TXT
TAG POS=R{{POS_Pix}} TYPE=IMG ATTR=SRC:https://*.fbcdn.net/*_n.jpg* EXTRACT=TXT
SET !EXTRACT {{!LOOP}}
TAG POS=R1 TYPE=A ATTR=HREF:*?ref=br_rs EXTRACT=TXT
TAG POS=R5 TYPE=A ATTR=HREF:*?ref=br_rs EXTRACT=TXT
SAVEAS TYPE=EXTRACT FOLDER=* FILE=file.csv
WAIT SECONDS=1
i tested this with 20 profiles. out of those profiles only 12 had city and state data present. out of 12 this script was only able to extract 8. the code is not finessed enough to hit 100% of the city,state because i simply do not know how to do that. so right now it is about a 66% hit rate. perhaps if i scraped 100k users i should expect to only have 66k legitimate users with data that i can use. and i used your extract loop method. so if the script can't find tag anymore i can stop it, input the right loop number, and fire it up again. with that being said, this script should be able to scrape any facebook page's likes. the data can then be formatted in excel automatically to list: first name, last name, city, state. then input the info to facebook ads and that is your custom audience. the only issue is that the second column of data where city,state is in there is a jumble of random data that is NOT city,state such as high school or university names. this is the nature of the script, it will just scrape whatever is in R5. so, i'm not sure how that will effect building the audience in facebook ads, perhaps it won't matter and those users with jumbled data won't be matched anyway

if you would like to clean up this current script to make it more accurate that would be great, thanks for the assistance
chivracq
Posts: 8198
Joined: Sat Apr 13, 2013 1:07 pm
Location: Amsterdam (NL)

Re: How to increase TAG POS by 4 automatically?

Post by chivracq » Mon May 28, 2018 1:32 am

PBNSurprise wrote:hey thanks for the reply, i will go ahead and test that. can you please let me know how the relative positioning works for tag POS? this is the code that i am working with now:

Code: Select all

VERSION BUILD=8820413 RECORDER=FX
SET !ERRORIGNORE YES
SET !EXTRACT_TEST_POPUP NO
TAB T=1

SET Like_Search "Fazio's Cat Jewelry"
SET POS_Pix {{!LOOP}}

'TAG POS={{!LOOP}} TYPE=DIV ATTR=TXT:Likes<SP>{{Like_Search}}*

TAG POS=1 TYPE=DIV ATTR=TXT:Add<SP>FriendFriend<SP>request<SP>sentMore<SP>*
TAG POS=R-1 TYPE=* ATTR=* EXTRACT=TXT
TAG POS=R-4 TYPE=* ATTR=* EXTRACT=TXT
TAG POS=R{{POS_Pix}} TYPE=IMG ATTR=SRC:https://*.fbcdn.net/*_n.jpg* EXTRACT=TXT
SET !EXTRACT NULL
TAG POS=R1 TYPE=A ATTR=HREF:*?ref=br_rs EXTRACT=TXT
TAG POS=R4 TYPE=A ATTR=HREF:*/pages/*?ref=br_rs EXTRACT=TXT
SAVEAS TYPE=EXTRACT FOLDER=* FILE=file.csv
WAIT SECONDS=1
it successfully scrapes two columns of data, however the second column is jumbled up with city,state and other data that i don't need. this is probably because my TAG POS=R is set incorrectly. i tried using position 4 and that works sometimes, but it seems the relative position of city,state for each profile changes. is this difficult to get around or is this an easy fix? sorry but i don't understand imacros quite well, i do believe that the issue lies within the TAG POS=R though, correctly identifying the correct relative position for city,state and scraping the data so that it matches the name. i'm going to continue to fiddle with this to see if it's something easy to solve but last resort i will may just go to upwork and hire a professional. if this is an easy fix please let me know so i can implement it right away and begin scraping, thanks!

EDIT: i think a better solution would be for me to figure out how to use tag relative position to target the last line of text for each profile
Image
that last line is where the city,state info will always be if it's available. will try to see how i can target that
PBNSurprise wrote:well i fiddled with it for a bit and this is what i came up with:

Code: Select all

VERSION BUILD=8820413 RECORDER=FX
SET !ERRORIGNORE YES
SET !EXTRACT_TEST_POPUP NO
TAB T=1

SET !LOOP 1
SET POS_Pix {{!LOOP}}

'TAG POS={{!LOOP}} TYPE=DIV ATTR=TXT:Likes<SP>{{Like_Search}}*

TAG POS=1 TYPE=DIV ATTR=TXT:Add<SP>FriendFriend<SP>request<SP>sentMore<SP>*
TAG POS=R-1 TYPE=* ATTR=* EXTRACT=TXT
TAG POS=R-5 TYPE=* ATTR=* EXTRACT=TXT
TAG POS=R{{POS_Pix}} TYPE=IMG ATTR=SRC:https://*.fbcdn.net/*_n.jpg* EXTRACT=TXT
SET !EXTRACT {{!LOOP}}
TAG POS=R1 TYPE=A ATTR=HREF:*?ref=br_rs EXTRACT=TXT
TAG POS=R5 TYPE=A ATTR=HREF:*?ref=br_rs EXTRACT=TXT
SAVEAS TYPE=EXTRACT FOLDER=* FILE=file.csv
WAIT SECONDS=1
i tested this with 20 profiles. out of those profiles only 12 had city and state data present. out of 12 this script was only able to extract 8. the code is not finessed enough to hit 100% of the city,state because i simply do not know how to do that. so right now it is about a 66% hit rate. perhaps if i scraped 100k users i should expect to only have 66k legitimate users with data that i can use. and i used your extract loop method. so if the script can't find tag anymore i can stop it, input the right loop number, and fire it up again. with that being said, this script should be able to scrape any facebook page's likes. the data can then be formatted in excel automatically to list: first name, last name, city, state. then input the info to facebook ads and that is your custom audience. the only issue is that the second column of data where city,state is in there is a jumble of random data that is NOT city,state such as high school or university names. this is the nature of the script, it will just scrape whatever is in R5. so, i'm not sure how that will effect building the audience in facebook ads, perhaps it won't matter and those users with jumbled data won't be matched anyway

if you would like to clean up this current script to make it more accurate that would be great, thanks for the assistance
Hum, so you want to scrap more than just the Name of those Users, I'm a bit against that, if you understood, I normally never help for FB and Social Media..., is "Fake" Use of Social Media to me, or leads to Spamming, and I'm totally against that, sorry, I've already helped you "too much", ah-ah...!

You are doing quite well though to be honest, your 2nd Script is pretty close, based on my latest Script, even if you didn't understand it "completely" and you "screwed" a bit one Mechanism with your Edit, ah-ah...! :idea:

But OK, good luck with your "i will may just go to upwork and hire a professional.", I would think I probably know all/most the "Professionals" able to "do the Job", you may want to post their Name/Link in this Thread and I can give you my "Rating" ah-ah...!, don't be fooled, many are a bit "Fake", and hum, I'm definitely an Advanced iMacros User I think but I would never defined myself as a "Professional", ah-ah...! :twisted:
(Hum, I actually do it a little bit on LinkedIn, but only because I cannot take part in any "serious" Tech Conversation otherwise once People see that I'm just "a DJ and Artist"...)
- (F)CIM = (Full) Config Info Missing: iMacros + Browser + OS with all 3 Versions...
- I usually don't even read the Question if that (required) Info is not mentioned...
- Script & URL usually help a lot for a more "educated" Help...
PBNSurprise
Posts: 15
Joined: Sun Jun 19, 2016 12:10 am

Re: How to increase TAG POS by 4 automatically?

Post by PBNSurprise » Mon May 28, 2018 3:45 am

thank you for the assistance thus far. i went ahead and scraped about 2.2k users before my firefox crashed and now i am manually formatting this data. i think by the end of it i will be left with a little over 1k users. i can't be bothered to continue scraping from this facebook page, for some reason i cannot load users as quickly on the infinite scroll page so this has become increasingly difficult. with that being said, i will settle with 1k users and then make a look-a-like audience from it. then i will target the look-a-like and the original 1k users. hopefully that will be enough of an advantage for me to effectively target my advertisements. we will see if the results come in and if the effort for all this was worth it
chivracq
Posts: 8198
Joined: Sat Apr 13, 2013 1:07 pm
Location: Amsterdam (NL)

Re: How to increase TAG POS by 4 automatically?

Post by chivracq » Mon May 28, 2018 5:10 am

PBNSurprise wrote:thank you for the assistance thus far. i went ahead and scraped about 2.2k users before my firefox crashed and now i am manually formatting this data. i think by the end of it i will be left with a little over 1k users. i can't be bothered to continue scraping from this facebook page, for some reason i cannot load users as quickly on the infinite scroll page so this has become increasingly difficult. with that being said, i will settle with 1k users and then make a look-a-like audience from it. then i will target the look-a-like and the original 1k users. hopefully that will be enough of an advantage for me to effectively target my advertisements. we will see if the results come in and if the effort for all this was worth it
Hum, poor 1k-2k Users that will then be spammed, oops...! :shock:

But hum, "before my firefox crashed", you were supposed to use IE from your FCI... :? (hum, that you never managed to mentioned completely btw, ah-ah...)
Check my own FCI that I must have mentioned after one of my Scripts..., mine doesn't crash ah-ah...!, and I regularly do much more "extreme" things than your Scrapping of 100K Users on some Site, you don't want to know, oops...! (And I usually only reboot my Laptop once every 3 weeks, just to make "sure"..., unless I have a Gig somewhere (I'm a DJ thus IRL...))
- (F)CIM = (Full) Config Info Missing: iMacros + Browser + OS with all 3 Versions...
- I usually don't even read the Question if that (required) Info is not mentioned...
- Script & URL usually help a lot for a more "educated" Help...
Post Reply