Results 1 to 7 of 7

Thread: Scrape html with js and php?

  1. #1

    Thread Starter
    Addicted Member JackIlPazzo's Avatar
    Join Date
    Oct 2014
    Posts
    183

    Scrape html with js and php?

    Hello to all,

    Last week I started work on a parser in php to take the data from soccerway.com, the problem is that php as well as being a server-side language (thus some content that you upload later I can not pick them up), it is also very bare to parser level. The few that there are are full of bugs and slow, and in my case with simple_html_dom you eat a lot of memory on the server because it is totally wrong.
    So I'm going to write a parser in Javascript for the website at the top. I'm going to base myself on the MVC model, the idea would be:

    1. A cron job php launches a JavaScript file that includes all classes with methods for each section which scrapare Site
    2. Any data collected from js is sent to php file that will take care of loading data on my database
    3. When an update is done to the log file

    What I want to ask you is if you can recommend a good parser in js (or a framework) that allows me to do this without much effort and in the manner as clean as possible, I do not know if you know CodeIgniter or Slim, these two frameworks provide methods for queries, so there is no need to write the query every time, but that's another story.

    I hope everything is clear

  2. #2
    MS SQL Powerposter szlamany's Avatar
    Join Date
    Mar 2004
    Location
    Connecticut
    Posts
    18,263

    Re: Scrape html with js and php?

    Just download the jQuery library and you will be traversing the DOM in JavaScript like there was no tomorrow!

    I've never used CodeIgniter or Slim - they might be more lightweight.

    *** Read the sticky in the DB forum about how to get your question answered quickly!! ***

    Please remember to rate posts! Rate any post you find helpful - even in old threads! Use the link to the left - "Rate this Post".

    Some Informative Links:
    [ SQL Rules to Live By ] [ Reserved SQL keywords ] [ When to use INDEX HINTS! ] [ Passing Multi-item Parameters to STORED PROCEDURES ]
    [ Solution to non-domain Windows Authentication ] [ Crazy things we do to shrink log files ] [ SQL 2005 Features ] [ Loading Pictures from DB ]

    MS MVP 2006, 2007, 2008

  3. #3
    Smooth Moperator techgnome's Avatar
    Join Date
    May 2002
    Posts
    34,532

    Re: Scrape html with js and php?

    CodeIgniter is a PHP-based MVC framework.

    Honestly... I'm struggling a bit on how this is supposed to work... For example the CRON job... you're expecting it to run some JS ... where? on the server? Is that even possible?

    -tg
    * I don't respond to private (PM) requests for help. It's not conducive to the general learning of others.*
    * I also don't respond to friend requests. Save a few bits and don't bother. I'll just end up rejecting anyways.*
    * How to get EFFECTIVE help: The Hitchhiker's Guide to Getting Help at VBF - Removing eels from your hovercraft *
    * How to Use Parameters * Create Disconnected ADO Recordset Clones * Set your VB6 ActiveX Compatibility * Get rid of those pesky VB Line Numbers * I swear I saved my data, where'd it run off to??? *

  4. #4
    MS SQL Powerposter szlamany's Avatar
    Join Date
    Mar 2004
    Location
    Connecticut
    Posts
    18,263

    Re: Scrape html with js and php?

    I'm with TG here - how are you going to get the external HTML from another site to be readable by the JavaScript running on your page?

    *** Read the sticky in the DB forum about how to get your question answered quickly!! ***

    Please remember to rate posts! Rate any post you find helpful - even in old threads! Use the link to the left - "Rate this Post".

    Some Informative Links:
    [ SQL Rules to Live By ] [ Reserved SQL keywords ] [ When to use INDEX HINTS! ] [ Passing Multi-item Parameters to STORED PROCEDURES ]
    [ Solution to non-domain Windows Authentication ] [ Crazy things we do to shrink log files ] [ SQL 2005 Features ] [ Loading Pictures from DB ]

    MS MVP 2006, 2007, 2008

  5. #5
    Frenzied Member tr333's Avatar
    Join Date
    Nov 2004
    Location
    /dev/st0
    Posts
    1,605

    Re: Scrape html with js and php?

    Perhaps you might want to look at Node.js then? There's a few node modules available for doing web scraping.
    CSS layout comes in to the 21st century with flexbox!
    Just another Perl hacker,

  6. #6

    Thread Starter
    Addicted Member JackIlPazzo's Avatar
    Join Date
    Oct 2014
    Posts
    183

    Re: Scrape html with js and php?

    For run javascript code from php is at least a php extension that embed (or wrap arround) a Javascript engine, and as a consequence, allows one to execute Javascript on the server, from PHP.
    The extension I'm thinking about is the spidermonkey
    I'm going to install it on the server, enabling it and will allow me to execute Javascript code, on the server, from PHP.
    Of course, like any other PHP extension, I need to be admin of the server, in order to install it.

    My goal as I said is use curl with php, pass the html grabbed to js and deserialize it. Some idea?

  7. #7
    Frenzied Member tr333's Avatar
    Join Date
    Nov 2004
    Location
    /dev/st0
    Posts
    1,605

    Re: Scrape html with js and php?

    If you want javascript on the server, just go straight to Node.js?
    CSS layout comes in to the 21st century with flexbox!
    Just another Perl hacker,

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  



Click Here to Expand Forum to Full Width