What programming languages would you use for this example web development project? What programming languages would you use for this example web development project?
Page 1 of 3 123 LastLast
Posts 1 to 10 of 25
  1. #1

    Nervous Newbie


    Join Date
    Jan 2019
    Posts
    5

    Question What programming languages would you use for this example web development project?

    Technically speaking, what is the best way of doing the project below? What language, technologies would you use and why please ?

    Summary as follows:

    • Scrape a lot of data (text, pictures, videos) from multiple websites multiple times in a day (As an example: all vehicle data from autotrader.co.uk)
    • Combine all data and check for any changes in the data being scraped (e.g new pictures)
    • Make data available on a website on a real time basis (database?)
    • Data is searchable with multiple filters (think autotrader.co.uk again)



    Many thanks

  2. #2

    Some things in Moderation

    cojak's Avatar
    Join Date
    Jul 2005
    Location
    Look to your right...
    Posts
    20,422

    Default

    Not even a 'please'?

    Edit - Oh sorry, there was a please at the end.

    I'm sure someone will be along in a minute who's keen to prep you on your interview...
    "I can put any old tat in my sig, put quotes around it and attribute to someone of whom I've heard, to make it sound true."
    - Voltaire/Benjamin Franklin/Anne Frank...

  3. #3

    More time posting than coding


    Join Date
    May 2018
    Posts
    358

    Default

    I'll give you a few thoughts for free

    If you want to compare images to see if they are different (for example if the front of the car is always called front.jpg and the dealer replaces the old one with a new one) you can get a hash of the image and compare that. You could exploit this idea to compare text or something as well.

    I'd also look into using already available software to turn HTML docs into json and store that in the database. That way, data from autotrader can be kept in the same table as data from Mumsnet. Databases like Cosmodb or Snowflake can query the json documents so you could have the same search form compare engine sizes from autotrader or diaper sizes on mumsnet.

  4. #4

    Double Godlike!

    malvolio's Avatar
    Join Date
    Jul 2005
    Location
    Walking in the garden, dreaming of Olivia...
    Posts
    11,883

    Default

    Just wondering why you would want to build a meta-serach engine for a site that already has a search engine...

    Anyway, we wrote something similar many years ago to combine a host of CMDBs to create an overall picture of the estate. We used C++ frontending the various DBMS tools already in place.

    And no, I don't have the code...
    Blog? What blog...?

  5. #5

    Richer than sasguru

    DimPrawn's Avatar
    Join Date
    Jul 2005
    Location
    Covid City
    Posts
    35,701

    Default

    VB6

    SARS-CoV-2 - Finally something from China that still works as designed.

  6. #6

    Super poster

    woohoo's Avatar
    Join Date
    Nov 2007
    Location
    In the country
    Posts
    4,759

    Default

    dBase III

  7. #7

    Prof Cunning @ Oxford Uni

    WTFH's Avatar
    Join Date
    Nov 2012
    Posts
    20,686

    Default

    Lisp.
    Practice social posting
    Stay Alert
    - Moderate the Posts -Save Face
    Together we can beat the CV offenders

  8. #8

    Respect my authoritah!

    NotAllThere's Avatar
    Join Date
    Aug 2007
    Location
    Far away from HMRC
    Posts
    22,817

    Default

    Clojure
    Down with racism. Long live miscegenation!

  9. #9

    More fingers than teeth

    BlasterBates's Avatar
    Join Date
    Jul 2005
    Location
    1922 Commitee
    Posts
    14,685

    Default

    The correct answer is python because parsing html comes out of the box. Yes it needs to go into a database as it should be searched. For images you can compare the size or even the name.

    Sounds like a website that shows all the results of all the other websites doing the same thing. Copyright might be a problem there unless it's hosted in some far flung lawless land.
    I'm alright Jack

  10. #10

    Default

    In the helpful spirit shown above: FORTRAN IV.
    When the fun stops, STOP.

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •