Skip to content

Python class to scrape data from rightmove.co.uk and return listings in a pandas DataFrame object

License

Notifications You must be signed in to change notification settings

toby-p/rightmove_webscraper.py

Repository files navigation

rightmove-webscraper

Downloads

rightmove.co.uk is one of the UK's largest property listings websites, hosting thousands of listings of properties for sale and to rent.

rightmove_webscraper.py is a simple Python interface to scrape property listings from the website and prepare them in a Pandas dataframe for analysis.

Installation

Version 1.1 is available to install via Pip:

pip install -U rightmove-webscraper

Scraping property listings

  1. Go to rightmove.co.uk and search for whatever region, postcode, city, etc. you are interested in. You can also add any additional filters, e.g. property type, price, number of bedrooms, etc.

  1. Run the search on the rightmove website and copy the URL of the first results page.

  2. Create an instance of the class with the URL as the init argument.

from rightmove_webscraper import RightmoveData

url = "https://www.rightmove.co.uk/property-for-sale/find.html?searchType=SALE&locationIdentifier=REGION%5E94346"
rm = RightmoveData(url)

What will be scraped?

When a RightmoveData instance is created it automatically scrapes every page of results available from the search URL. However please note that rightmove restricts the total possible number of results pages to 42. Therefore if you perform a search which could theoretically return many thousands of results (e.g. "all rental properties in London"), in practice you are limited to only scraping the first 1050 results (42 pages * 25 listings per page = 1050 total listings). A couple of suggested workarounds to this limitation are:

  • Reduce the search area and perform multiple scrapes, e.g. perform a search for each London borough instead of 1 search for all of London.
  • Add a search filter to shorten the timeframe in which listings were posted, e.g. search for all listings posted in the past 24 hours, and schedule the scrape to run daily.

Finally, note that not every piece of data listed on the rightmove website is scraped, instead it is just a subset of the most useful features, such as price, address, number of bedrooms, listing agent. If there are additional data items you think should be scraped, please submit an issue or even better go find the xml path and submit a pull request with the changes.

Accessing data

The following instance methods and properties are available to access the scraped data.

Full results as a Pandas.DataFrame

rm.get_results.head()
price type address url agent_url postcode full_postcode number_bedrooms search_date
0 3400000.0 2 bedroom apartment for sale Switch House East, Battersea Power Station, SW11 http://www.rightmove.co.uk/properties/121457195#/?channel=RES_BUY http://www.rightmove.co.uk/estate-agents/agent/JLL/London-Residential-Developments-100183.html SW11 NaN 2.0 2022-03-24 09:40:13.769706
1 11080000.0 Property for sale Battersea Power Station, Circus Road East, London http://www.rightmove.co.uk/properties/118473812#/?channel=RES_BUY http://www.rightmove.co.uk/estate-agents/agent/Moveli/London-191324.html NaN NaN NaN 2022-03-24 09:40:13.769706
2 9950000.0 5 bedroom apartment for sale 888 Scott House, Battersea Power Station, SW11 http://www.rightmove.co.uk/properties/89344718#/?channel=RES_BUY http://www.rightmove.co.uk/estate-agents/agent/Prestigious-Property-Ltd/Ruislip-67965.html SW11 NaN 5.0 2022-03-24 09:40:13.769706
3 9200000.0 3 bedroom penthouse for sale Battersea Power Station, Nine Elms, London SW8 http://www.rightmove.co.uk/properties/114236963#/?channel=RES_BUY http://www.rightmove.co.uk/estate-agents/agent/Copperstones/London-82091.html SW8 NaN 3.0 2022-03-24 09:40:13.769706
4 9000000.0 6 bedroom apartment for sale Scott House, Battersea Power Station, SW11 http://www.rightmove.co.uk/properties/107110697#/?channel=RES_BUY http://www.rightmove.co.uk/estate-agents/agent/Dockleys/London-174305.html SW11 NaN 6.0 2022-03-24 09:40:13.769706

Average price of all listings scraped

rm.average_price

1650065.841025641

Total number of listings scraped

rm.results_count

195

Summary statistics

By default shows the number of listings and average price grouped by the number of bedrooms:

rm.summary()
number_bedrooms count mean
0 0 39 9.119231e+05
1 1 46 1.012935e+06
2 2 88 1.654237e+06
3 3 15 3.870867e+06
4 4 2 2.968500e+06
5 5 1 9.950000e+06
6 6 1 9.000000e+06

Alternatively group the results by any other column from the .get_results DataFrame, for example by postcode:

rm.summary(by="postcode")
postcode count mean
0 SW11 76 1.598841e+06
1 SW8 28 2.171357e+06

Legal

@toddy86 has pointed out per the terms and conditions here the use of webscrapers is unauthorised by rightmove. So please don't use this package!