We already pull data from 29 different governance forums, including Arbitrum. Most of them are done via API, with appropriate rate limit handling, utilising proxies where appropriate, with efficient handling of all relevant associated data (post, author, etc). 
For our generalised scraper, we built a separate service that can crawl any web page and effectively pull out the text content of it to be aggregated in our pipeline. 
Currently our pipeline operates on a 4 hourly basis (so we crawl all data sources once every 4 hours), but this can be adjusted as needed. We are planning to get as close to real time as possible eventually.