Gonna take a break for a day or so. Something is going on with the web site? Don't want to waste time, so I will wait for a bit to make sure.

Splitting out to separate plugin today. After bugs cleanup, maybe that will resolve some of the other issues, such as memory crashes. Yeah, I can hear you thinking ... should have done it long ago.   

Gunnar, after being reminded of CefSharp, I downloaded and took another look today. Dead fast, easy, yummy.

The difference is already apparent from simply right-clicking on show page source. After clicking on the credit URL in the embedded IE browser, you can't see the info.

In Chrome on the desktop, you get the dreaded "confirm res-submission".

But in CefSharp, you see the entire credit detail HTML division, ripe for the scraping. Be still my heart.

Unless, you are already in progress, or intend to be, I propose to drop CLTBoss, at least temporarily and see how fast I can cobble up a quick CLT scrape. No plugin needed.

I would happily turn over the unpolished code to you, just to save you the trouble of slogging through the scraping. You could then, if you would like to, include the scrape directly in your tool and "Goodbye GoodGuy".

Now all CLTBoss needs to do simply grab XML from the online, because of CefSharp can deliver the complete profile ID without the messy scrape do-do I have been in, then we are sailing on to glory.

So what do you say? I will be spending the day seeing how fast a scrape I can cobble up, by porting over my logic from CLTBoss. If I can get to the credit detail, it is game over.

Quoting GSyren:

Quote:
With just screenscraping you can't get all the data that CLTBoss was extracting, right? So this would pretty much just return what CLTPlus did, but not give us anything more? No full profiles?

Right, but here is what I was thinking:

(1) People would like to have a replacement for CLTPlus - that could be CLTInfo

(2) I would still work on CLTBoss, but now its scraping would be optional, and use the same CefSharp code (if I get it working).

The biggest, and highly useful difference is that CLTInfo would be a complete tool, with no dependencies that could replace and go beyond CLTPlus. As a big added plus, it could also load XML that has been downloaded from the online, and do even more glorious statistics.

Shaking in my boots. Figured it out. Afraid to hope beyond hope, but it looks promising. Still requires a small delay (500 ms), but I have been able to click on a page and see it change, then click on a credits link, wait 500 ms and then grab the browser contents for scraping. Simple  tests, but it was fast and reliable.

If you guys are still tolerating my old muddle-brained antics, I am going to take a few hours of think time, have dinner and wine with my wife, and then I will cobble together the most basic, fastest "quick-scrape" that I can come up with. Then it will be available to anybody who wants it, and I will also retrofit it into CLTBoss.

Thanks for the detailed reply. I will go ahead and put the results into CLTBoss, if I am successful.

I think I got around the memory issue without using zip compression. Still testing.

Meanwhile I am retrofitting the XPath search to the minimalist CLTBoss XML, and I have gotten myself confused about group and episode dividers.

Correct me if I am wrong, but are the groups and episode designations not available in the XML, or am I missing something? I am trying to avoid losing the advantages of the Xpath by needing to manually loop through all the credits anyway, just to correctly identify groups and episodes.

Can someone help clear my foggy brain. I am so loopy from all the rewrites and testing, etc that I have forgotten some stuff I ought to know. 

Geez, I must be really losing it. Last night I looked at what I thought was an XML sample that should have contained those entries and got me all loopy. Today is another day. Thanks!

I've usually parsed divider credits only from the API, not the XML, but XPath works on the XML and when the guts of the credit are revealed to me in the XPath extension, dividers are already gone. Tricky.