I used to use a combo of python tools. Requests, beautifulsoup mostly. However the last few things I've built used selenium to drive headless chrome browsers. This allows me to run the javascript most sites use these days.