jameda - Extract data of all physicians

Discussions and Tech Support related to website data extraction, screen scraping and data mining using iMacros.
Forum rules
iMacros EOL - Attention!

The renewal maintenance has officially ended for Progress iMacros effective November 20, 2023 and all versions of iMacros are now considered EOL (End-of-Life). The iMacros products will no longer be supported by Progress (aside from customer license issues), and these forums will also no longer be moderated from the Progress side.

Thank you again for your business and support.

Sincerely,
The Progress Team

Before asking a question or reporting an issue:
1. Please review the list of FAQ's.
2. Use the search box (at the top of each forum page) to see if a similar problem or question has already been addressed.
3. Try searching the iMacros Wiki - it contains the complete iMacros reference as well as plenty of samples and tutorials.
4. We can respond much faster to your posts if you include the following information: CLICK HERE FOR IMPORTANT INFORMATION TO INCLUDE IN YOUR POST
Post Reply
i_macro_user
Posts: 76
Joined: Sun Dec 02, 2012 5:28 pm

jameda - Extract data of all physicians

Post by i_macro_user » Tue Aug 28, 2018 5:11 pm

Hallo

https://www.jameda.de/search/berlin/aer ... achgebiet/

How can I use iMacros 12 on Windows 8 to extract the following data from all physicians on the above website in a loop using the iMacros Browser?

I need

name
telephone number
homepage
Office hours from Monday to Sunday

To view this data, click on the "View profile" button for each data record from the overview page.

To load all data, click on "Show more" at the end of each page.
How do I know that there is a "Show more" button?
When are all data records saved?

Would any of you like to write an example of a loop for exactly this interset page?

The data should be saved in a csv file.

Thank you very much.
Last edited by i_macro_user on Tue Aug 28, 2018 6:16 pm, edited 1 time in total.
Hum ... Only professional answers are helpful
Hum ... No complete quotations - limit to the essentials
Hum ... Can we get along in this forum without personal attacks?
Hum ... I use automatic translations
Yes ... I am iMacros beginners :-)
chivracq
Posts: 10301
Joined: Sat Apr 13, 2013 1:07 pm
Location: Amsterdam (NL)

Re: jameda

Post by chivracq » Tue Aug 28, 2018 5:38 pm

i_macro_user wrote:Hallo

https://www.jameda.de/search/berlin/aer ... achgebiet/

How can I use

Code: Select all

iMacros 12 on Windows 8
to extract the following data from all physicians on the above website in a loop using the iMacros Browser?

I need

name
telephone number
homepage
Office hours from Monday to Sunday

To view this data, click on the "View profile" button for each data record from the overview page.

To load all data, click on "Show more" at the end of each page.
How do I know that there is a "Show more" button?
When are all data records saved?

Would any of you like to write an example of a loop for exactly this interset page?

The data should be saved in a csv file.

Thank you very much.
Hum..., "jameda" as a Thread Title is not very-very explicit/descriptive, I understand that's the Name of a Site, but I guess you could have found a little bit more "explicit" Title..., for other Users searching the Forum and who could be interested to (re)use any Content/Technique/Script from this Thread for maybe some other Site(s)...

OK, good to see that you've finally "understood" that English is the Forum Language, and dropped German..., Reason why I stopped answering your few last previous Threads, tja...! :D

Good that you mention your FCI (or I wouldn't be reacting at all...!), but when using iMB v12.0, you need to mention if you are using the 'Trial' or 'Full/Enterprise' Edition...?
And you should edit your Sig which still mentions your FCI from 4 years ago apparently..., this is now 4 years later a bit "confusing"...:

Code: Select all

iMacros 10, Windows 8 / 64, I use the iMacros Browser (not MSIE) - sometimes Firefox - every time the last version - 2014-12-10
EDIT: Ah OK, you've modified your Sig already in the meantime... But hum, "Beginner", you've been using iMacros since 2012, you are not a 'Beginner" anymore, ah-ah...! :roll:

>>>

Hum..., OK..., I'm "eventually" willing to have a look at this Site..., but hum..., I won't nor anybody else will be writing your Script, good luck with that, ah-ah...! Nice Try...! :shock:
You need to do a bit "your part of the job"..., what is the Pb/Difficulty...?, what have you tried and where do you get stuck...? :?:
- (F)CI(M) = (Full) Config Info (Missing): iMacros + Browser + OS (+ all 3 Versions + 'Free'/'PE'/'Trial').
- FCI not mentioned: I don't even read the Qt...! (or only to catch Spam!)
- Script & URL help a lot for more "educated" Help...
i_macro_user
Posts: 76
Joined: Sun Dec 02, 2012 5:28 pm

Re: jameda - Extract data of all physicians

Post by i_macro_user » Tue Aug 28, 2018 6:52 pm

I have adapted the title of the article. Thanks for pointing that out.

I'm a beginner - I've never managed to extract large amounts of data from websites before.

Although iMacros is already available in version 12, automatic recording of loops is not possible. The entry hurdle is very high.

(1)

The first record is opened with:

TAG POS=1 TYPE=SPAN ATTR=TXT:Profil<SP>ansehen
'New tab opened
TAB T=2

How do I open all other data records on the page with the loop variable?

How do I know how many records are on the page?

How do I know which record I need to click on "Show more"?

TAG POS=1 TYPE=BUTTON:SUBMIT ATTR=TXT:Mehr<SP>anzeigen

How do I know that there are no more records?

(2)

Try extracting the phone number on the profile page.

Problem A

the position of the telephone number is always different for each profile

TAG POS=76 TYPE=DIV ATTR=* EXTRACT=TXT
TAG POS=84 TYPE=DIV ATTR=* EXTRACT=TXT

Problem B

Not only the phone number is extracted, but a lot more text:

"Phone number" and the text

---------------------------
iMacros
---------------------------
Extracted text:

030/xxx3788noch nicht hinterlegtSie sind Dr. Schüler? Buchen Sie jetzt ein jameda Premium-Paket um diese und viele weitere Funktionen nutzen zu können.

---------------------------
OK
---------------------------

As you can see, the extracted phone number also contains text and the name of the physician. Every time the doctor's name changes, I don't know how I can separate the phone number from the following text with iMacros before saving.
IMacros displays the above extract command as the smallest unit of data to extract.

How do you extract the phone number?

And so there are many other questions on which I do not continue, and that is why I am asking for help here in the forum.

I can write a whole novel of questions, but does that help in any way?

If it were a simple database to extract, it would certainly be possible.
Anyone who looks at the website will notice that this is a weightless one.



( Conclusion )

Questions over questions.

For someone who knows iMacros well, it's easy to write the script you need.

Hence my request in this forum.

Everyone can learn best from examples because they are easy to understand.

Step by step.

I know you have the expertise.
Hum ... Only professional answers are helpful
Hum ... No complete quotations - limit to the essentials
Hum ... Can we get along in this forum without personal attacks?
Hum ... I use automatic translations
Yes ... I am iMacros beginners :-)
chivracq
Posts: 10301
Joined: Sat Apr 13, 2013 1:07 pm
Location: Amsterdam (NL)

Re: jameda - Extract data of all physicians

Post by chivracq » Wed Aug 29, 2018 12:33 am

i_macro_user wrote:I have adapted the title of the article. Thanks for pointing that out.

I'm a beginner - I've never managed to extract large amounts of data from websites before.

Although iMacros is already available in version 12, automatic recording of loops is not possible. The entry hurdle is very high.

(1)

The first record is opened with:

Code: Select all

TAG POS=1 TYPE=SPAN ATTR=TXT:Profil<SP>ansehen
'New tab opened
TAB T=2
How do I open all other data records on the page with the loop variable?

How do I know how many records are on the page?

How do I know which record I need to click on "Show more"?

Code: Select all

TAG POS=1 TYPE=BUTTON:SUBMIT ATTR=TXT:Mehr<SP>anzeigen
How do I know that there are no more records?

(2)

Try extracting the phone number on the profile page.

Problem A

the position of the telephone number is always different for each profile

Code: Select all

TAG POS=76 TYPE=DIV ATTR=* EXTRACT=TXT
TAG POS=84 TYPE=DIV ATTR=* EXTRACT=TXT
Problem B

Not only the phone number is extracted, but a lot more text:

"Phone number" and the text

---------------------------
iMacros
---------------------------
Extracted text:
030/xxx3788noch nicht hinterlegtSie sind Dr. Schüler? Buchen Sie jetzt ein jameda Premium-Paket um diese und viele weitere Funktionen nutzen zu können.
---------------------------
OK
---------------------------

As you can see, the extracted phone number also contains text and the name of the physician. Every time the doctor's name changes, I don't know how I can separate the phone number from the following text with iMacros before saving.
IMacros displays the above extract command as the smallest unit of data to extract.

How do you extract the phone number?

And so there are many other questions on which I do not continue, and that is why I am asking for help here in the forum.

I can write a whole novel of questions, but does that help in any way?

If it were a simple database to extract, it would certainly be possible.
Anyone who looks at the website will notice that this is a weightless one.

( Conclusion )
Questions over questions.

For someone who knows iMacros well, it's easy to write the script you need.

Hence my request in this forum.

Everyone can learn best from examples because they are easy to understand.

Step by step.

I know you have the expertise.
Yep, Thread Title is indeed a bit better now..., Thanks... :D

Hum, if your "Definition" of a "Beginner" is "I'm a beginner - I've never managed to extract large amounts of data from websites before.", then-then-then..., oops, I'm a complete Beginner then, because I've never tried...! And though I know I will succeed if I "try"..., ah-ah...!
=> That's a bit of a stupid Answer, sorry...! :shock: You've been using iMacros since at least 2012, using the Professional iMB v10 Edition for many years (I've never used any iMB Version myself), with a few Dozens Threads on the Forum since 2012, and you still call yourself a Beginner, come on...! :roll:

Oh...!, but hum, I asked you to mention if you were using iMB v12.0 'Trial'/'Enterprise' but you didn't react to that... A bit of a Show-Stopper for me, hum..., 'Trial' I guess, or you would be asking your Qt to TechSup directly and not on the Forum, I reckon... I guess you've been using the Trial Version (=30 days...!) for 6 years already, ah-ah...!
But OK, so you didn't answer my Qt about your FCI..., :roll: OK-OK, then I'll just answer/react (to) a few "easy" parts... (I still didn't have a look at your Site...)

Yeah, well, all your Qt's like "How do I open all other data records..." / "How do I know how many records..." / "How do I know which record..." etc are indeed Newbie Qt's of sbd who never tried anything and who didn't read any Documentation..., tja...! You have a complete Section in the Wiki about Data Extraction, read it and understand it and your Answers are already in there... (And you had plenty of time in the last 6 years to read the whole Wiki I would think, ah-ah...! "Question of Motivation"...! :idea: )

=> Same about the Tel_Nr's you want to extract with 'POS=76/84', yeah..., basic Extraction Technique with 'Relative Positioning', sorry, read a bit of the Documentation, you will learn 20x times more and quicker than from my stupid and useless "Answer"... :oops:

Do you best and make your FULL Script yourself and I'll be able to help you if you REALLY get stuck somewhere, then I will find a Solution for you, but you haven't hardly tried anything yourself... :idea:

OK, good "luck"... :wink:

>>>

[And don't worry about English not being your native Language, same like 80% of the Forum Users, me included, even if speak English/German/French among a few other Languages "a bit correctly", same like being an iMacros Users for at least 6 years, you had 6 years also to learn English a bit correctly then (all Documentation + the Forum are in English, ah-ah...!, you were for several years the only "weird Specimen" on the Forum who kept asking in German, ah-ah...! :shock: ), a bit of a useless Excuse in my Opinion, ah-ah...! Hum, even if I do have the Feeling your English improved a clear bit indeed in the last few years, must come from you, Google Translate is still completely shitty, ah-ah...! But pfff, we are talking about "Years"...! (=> x6...! :shock: ), it took me 3 Weeks to learn Dutch for example, and 3 Months to speak it fluently, tja...! Anybody can learn any Language in 6 Months if you/they are mini-bit motivated...! 6 years sounds a bit like a "Joke" to me to learn English..., especially if you are native German... :idea: ]

Hum, and maybe even from the 20% native English speaking Users on the Forum (at least from their IP-Address, which is no "Exact Science" either, nobody joining the Forum gets asked (luckily...!) about their Address and Nationality and native Language(s)), I for example am French natively but live in NL, so I have a Dutch IP, but I guess the "Stats" are still a little bit correct, at least "in the big Numbers statistically"..., => maybe only 50% (=> 10%) really speak "correct" English, with many Teens using SMS-Language and approx Grammar & Spelling, I think "Foreigners" often pay more attention to their Writing on some International Forum in English than People speaking natively English... (My Perception... :oops: )
- (F)CI(M) = (Full) Config Info (Missing): iMacros + Browser + OS (+ all 3 Versions + 'Free'/'PE'/'Trial').
- FCI not mentioned: I don't even read the Qt...! (or only to catch Spam!)
- Script & URL help a lot for more "educated" Help...
i_macro_user
Posts: 76
Joined: Sun Dec 02, 2012 5:28 pm

Re: jameda - Extract data of all physicians

Post by i_macro_user » Fri Aug 31, 2018 10:23 am

Why do you quote the complete previous text?
This makes the contributions more difficult to read.

I will be happy to answer your question - maybe you, too, with a solution to one of my questions? :-)

I tried version 12 as a trial in the hope that there are now automatisms for data extraction in a loop.

Let's try a single question consisting of 2 sections:

Please have a look at the link:

https://www.jameda.de/search/berlin/aer ... achgebiet/

(a) How do I know if there is "show more" at the bottom of the page?

(b) How do I know how many records are currently displayed?

What other ways are there to create an automatic loop to extract all records on a page if I don't know how many records are displayed?

__ offtopic __________________________________________________________

Another private remark, which I am sure you will resent.

Instead of clear answers there are Chivracq "sum" discussions.
Hardly anyone dares to tell you that, because you are effectively the only one who can give professional answers here.

The problem with this forum is that you are effectively the only one who can respond to topics with expertise if he wants. You often don't want to, because something personally bothers you about the questioner. In the past, iMacros employees responded more often in the forum - earlier.

Reading something in an English wiki is one side of the coin.

To understand and apply to one - in this case - concrete doctors database, the other side of the coin.

And the third - real side of the coin is: there are too few complete examples of concrete websites.
Much more can be learned from complete and well-documented examples.

For all programs, macros and script languages I use, there are a lot of well documented and complete scripts. Only iMacros has little or nothing.

This would be a real challenge and really helpful if you would publish complete sample scripts for specific websites.
And you could link to them if you have any questions.

Because the questions are always the same, from login to data from/to Excel with iMacros etc.

Examples for loops, databases, underlaid with youTube videos.

Do you seriously believe after my experience in this forum that I would ask questions if there was not an important and urgent (in this case medical) background?

And no - it's not about ophthalmologists, the medical background is sadder.

Look at the number of questions I ask year after year. Notice anything?

Starting with iMacros 7 years ago doesn't mean I'm not a beginner.

The barriers to entry are too high for me.
I'm a beginner.

No matter whether it is about the implementation of JavaScript, databases and more.

I now have a database of 142 open questions on iMacros from 7 years.

Instead of clear answers there are Chivracq "sum" discussions. Therefore, the questions remained offline and iMacros dusty. Again and again I check if there are other active iMacros forums - unfortunately no.

Or are there alternative iMacros forums?

I have used iMacros only very rarely (but I like it anyway), because the entry hurdles are too high and the support in the forum from iMacros and the few users is too low.

Have you ever wondered why the iMacros programmers haven't really managed to improve the program in 7 years?

It starts with the nonsense that you have to separate strings with <SP> instead of just putting them in quotation marks. And it still doesn't stop with the fact that there are still no iMacros in languages other than English. After countless years of development.
Many much more complex (and sometimes free) programs are available in different languages.

The good news: for many topics there are forums that are active and lively.
The bad news: this does not help with iMacros.
Hum ... Only professional answers are helpful
Hum ... No complete quotations - limit to the essentials
Hum ... Can we get along in this forum without personal attacks?
Hum ... I use automatic translations
Yes ... I am iMacros beginners :-)
i_macro_user
Posts: 76
Joined: Sun Dec 02, 2012 5:28 pm

Re: jameda - Extract data of all physicians

Post by i_macro_user » Fri Aug 31, 2018 10:44 am

brief supplement

https://forum.imacros.net/viewtopic.php?f=2&t=29735

PS: My English has not improved - I use better automatic translations.

PS: Can we get along in this forum without personal attacks?

Professional answers are helpful.
Hum ... Only professional answers are helpful
Hum ... No complete quotations - limit to the essentials
Hum ... Can we get along in this forum without personal attacks?
Hum ... I use automatic translations
Yes ... I am iMacros beginners :-)
chivracq
Posts: 10301
Joined: Sat Apr 13, 2013 1:07 pm
Location: Amsterdam (NL)

Re: jameda - Extract data of all physicians

Post by chivracq » Sat Sep 01, 2018 3:31 am

i_macro_user wrote:Why do you quote the complete previous text?
This makes the contributions more difficult to read.
Yep, completely agree, but I systematically quote, based on many Users deleting/truncating their Posts as they finally get their Script working, to prevent any "Competitors" on the same Site to use the same Script... :shock:
Only a very few "select" Users usually already (Semi-) Advanced and who have already proved to be using the Forum VERY correctly I won't quote systematically... Hum, you have an Example in this current thread with @thecoder2012..., ah-ah...! :wink:

But @OP you definitely don't belong to that "Pool" of "trustworthy" Users, ah-ah...! :lol: , still a very long way to go Darling...! Hum..., have to laugh a little bit, oops..! Thanks in a way... :D
(Hum, and mini-proactive Feedback, I'm afraid/glad you (=> @OP) will never belong to that "select" Group, ah-ah...!, from the way you've been using the Forum for so many years since you joined, tja...! :shock: )

Other Scenario is Users keeping editing their OP as they "progress", which makes such Threads useless to follow as well for other Users trying to understand how a Situation got solved with an Original Script from OP till Final Post with Working Script.. 8)
i_macro_user wrote:I will be happy to answer your question - maybe you, too, with a solution to one of my questions? :-)

I tried version 12 as a trial in the hope that there are now automatisms for data extraction in a loop.

Let's try a single question consisting of 2 sections:

Please have a look at the link:

https://www.jameda.de/search/berlin/aer ... achgebiet/

(a) How do I know if there is "show more" at the bottom of the page?

(b) How do I know how many records are currently displayed?

What other ways are there to create an automatic loop to extract all records on a page if I don't know how many records are displayed?
Yeah, still don't see any Script from you, I won't write your Script, sorry...! :roll:

>>>
i_macro_user wrote:__ offtopic __________________________________________________________

Another private remark, which I am sure you will resent.

Instead of clear answers there are Chivracq "sum" discussions.
Hardly anyone dares to tell you that, because you are effectively the only one who can give professional answers here.

The problem with this forum is that you are effectively the only one who can respond to topics with expertise if he wants. You often don't want to, because something personally bothers you about the questioner. In the past, iMacros employees responded more often in the forum - earlier.

Reading something in an English wiki is one side of the coin.

To understand and apply to one - in this case - concrete doctors database, the other side of the coin.

And the third - real side of the coin is: there are too few complete examples of concrete websites.
Much more can be learned from complete and well-documented examples.

For all programs, macros and script languages I use, there are a lot of well documented and complete scripts. Only iMacros has little or nothing.

This would be a real challenge and really helpful if you would publish complete sample scripts for specific websites.
And you could link to them if you have any questions.

Because the questions are always the same, from login to data from/to Excel with iMacros etc.

Examples for loops, databases, underlaid with youTube videos.

Do you seriously believe after my experience in this forum that I would ask questions if there was not an important and urgent (in this case medical) background?

And no - it's not about ophthalmologists, the medical background is sadder.

Look at the number of questions I ask year after year. Notice anything?

Starting with iMacros 7 years ago doesn't mean I'm not a beginner.

The barriers to entry are too high for me.
I'm a beginner.

No matter whether it is about the implementation of JavaScript, databases and more.

I now have a database of 142 open questions on iMacros from 7 years.

Instead of clear answers there are Chivracq "sum" discussions. Therefore, the questions remained offline and iMacros dusty. Again and again I check if there are other active iMacros forums - unfortunately no.

Or are there alternative iMacros forums?

I have used iMacros only very rarely (but I like it anyway), because the entry hurdles are too high and the support in the forum from iMacros and the few users is too low.

Have you ever wondered why the iMacros programmers haven't really managed to improve the program in 7 years?

It starts with the nonsense that you have to separate strings with <SP> instead of just putting them in quotation marks. And it still doesn't stop with the fact that there are still no iMacros in languages other than English. After countless years of development.
Many much more complex (and sometimes free) programs are available in different languages.

The good news: for many topics there are forums that are active and lively.
The bad news: this does not help with iMacros.
i_macro_user wrote:brief supplement

https://forum.imacros.net/viewtopic.php?f=2&t=29735

PS: My English has not improved - I use better automatic translations.

PS: Can we get along in this forum without personal attacks?

Professional answers are helpful.
=> Fair enough, I'll be happy to reply to your Qt's/Remarks/Complaints/Suggs..., but C&P that Content into some apart Thread (in the 'General' Sub-Forum), like you perfectly Legit'ly did with your "When does iMacros get better?"...

This last Quote like you realized yourself has nothing to do with this current Thread and even less with the 'Data Extraction' Sub-Forum... :wink:

Sig:
Hum ... Only professional answers are helpful
Hum ... No complete quotations - limit to the essentials
Hum ... Can we get along in this forum without personal attacks?
Hum ... I use automatic translations
Yes ... I am iMacros beginners :-)
Only quoting for now, might reply later/tomorrow... :wink:

=> EDIT: 1 day later, see my Comments after each separate Quotes..
Last edited by chivracq on Mon Sep 03, 2018 4:37 am, edited 9 times in total.
- (F)CI(M) = (Full) Config Info (Missing): iMacros + Browser + OS (+ all 3 Versions + 'Free'/'PE'/'Trial').
- FCI not mentioned: I don't even read the Qt...! (or only to catch Spam!)
- Script & URL help a lot for more "educated" Help...
User avatar
thecoder2012
Posts: 446
Joined: Sat Aug 15, 2015 5:14 pm
Location: Internet
Contact:

Re: jameda - Extract data of all physicians

Post by thecoder2012 » Sat Sep 01, 2018 7:23 am

i_macro_user wrote:030/xxx3788noch nicht hinterlegtSie sind Dr. Schüler? Buchen Sie jetzt ein jameda Premium-Paket um diese und viele weitere Funktionen nutzen zu können.
Vielleicht wäre es einfacher in Deutsch zu schreiben in diesem Fall. Das Englisch klingt nicht sinnvoll.
(Maybe it would be easier to write in German in this case. English doesn't make sense.)
i_macro_user wrote:How do you extract the phone number?
You can use javascript in a new tab and grab the results. Or make your imacros script better.
i_macro_user wrote:And so there are many other questions on which I do not continue, and that is why I am asking for help here in the forum.
Full example (and comments) with your problem is very helpful.
i_macro_user wrote:For someone who knows iMacros well, it's easy to write the script you need.
Not really without Firefox + iMacros 8.9.7 (or other version with good javascript support and features)
i_macro_user wrote:Everyone can learn best from examples because they are easy to understand.
Step by step.
I know you have the expertise.
Not for free without your full example.
Hum ... I use automatic translations
Only google translate? With bing translate, google translate and deepl.com as team it's not really bad in most cases with a little human knowledge in english.
Yes ... I am iMacros beginners :-)
We have better users in this forum? :lol:
Join 9kw.eu Captcha Service now and let your iMacros continue downloads and scripts while you sleep. - Custom iMacros? Contact me! :idea:
chivracq
Posts: 10301
Joined: Sat Apr 13, 2013 1:07 pm
Location: Amsterdam (NL)

Re: jameda - Extract data of all physicians

Post by chivracq » Mon Sep 03, 2018 4:41 am

thecoder2012 wrote:Vielleicht wäre es einfacher in Deutsch zu schreiben in diesem Fall. Das Englisch klingt nicht sinnvoll.
NEIN...! (Luschtik klinke ischt imma sinvol, ah-ah...! :wink: ) Forum-Language is English... :roll:
- (F)CI(M) = (Full) Config Info (Missing): iMacros + Browser + OS (+ all 3 Versions + 'Free'/'PE'/'Trial').
- FCI not mentioned: I don't even read the Qt...! (or only to catch Spam!)
- Script & URL help a lot for more "educated" Help...
Post Reply