Tons of bug fixes & little improvements & updated dependencies
What's Changed
- Added new project that uses Colly by @twiny in #525
- adding docker-slim to the list of open source projects using Colly by @kcq in #526
- update coursera example by @ysung6 in #537
- Update openedx_courses.go by @lingt-xyz in #539
- feat: support Stop() in queue by @PalanQu in #531
- Fix invalid base URL assignment by @dasshield in #549
- Add context.Context support by @WGH- in #548
- Fix handling of redirected URLs in OnResponseHeaders by @WGH- in #561
- Fix race on count fields of Collector type by @shubham14bajpai in #563
- SetProxyFunc: fix proxies rotation by @wowinter13 in #567
- Update xkcd_store.go by @BigManing in #570
- Updated xmlquery to close a known security vulnerability by @ijomeli in #582
- Fix default User-Agent when using custom headers by @WGH- in #592
- Update openedx_courses.go by @BigManing in #576
- Add httpBackend cache response callback by @slashapp in #574
- Update README.md by @seversky in #595
- Change the type assersion for Attr by @moritamori in #588
- Update User-Agent strings for Desktop by @guessi in #605
- add quote crawler using colly by @eval-exec in #603
- Fix golint issues by @waltton in #620
- Replace Travis with GitHub Actions by @alexhraber in #669
- Use github.com/nlnwa/whatwg-url for URL parsing by @WGH- in #673
- Fix Async ignoring its arguments by @ljbink in #652
- fix: generated code has compile error by @darjun in #629
- moving filterings to one function (checkFilters) and use it in case of redirect too by @joelazar in #638
- Fix Contributing.md link by @SaladinoBelisario in #641
- Work around invalid URL encoding in path by @WGH- in #675
- Host & Custom Header Support by @ErikOwen in #691
- Code cleanup by @WGH- in #678
- Add GitHub issue templates and nudge questions to Stack Overflow by @WGH- in #702
- Fix redirects ignoring AllowURLRevisit=false by @WGH- in #681
- Bump whatwg-url version by @WGH- in #713
- fix some typos by @cuishuang in #736
- feat: Add MaxRequests paramete by @guilhem in #722
- Update CI configuration by @WGH- in #749
- upgrade Cascadia version by @kinoute in #650
- Bump golang.org/x/net from 0.0.0-20220114011407-0dd24b26b47d to 0.7.0 by @dependabot in #756
- Support websites redirecting to the same page when AllowURLRevisit is disabled by @WGH- in #763
- Update User-Agent strings for 2023 by @guessi in #765
- Fix data races by @WGH- in #771
- Replaced deprecated io/ioutil package with proper implementations of … by @zachary-walters in #785
- Bump golang.org/x/net from 0.7.0 to 0.17.0 by @dependabot in #789
- put accept header before OnRequest by @k4lizen in #786
- Fix Bug: retry scrape will lost POST requestData by @Shinku-Chen in #794
- scrapper -> scraper Update README.md by @ideabrian in #809
- Test on Go 1.22, drop old versions by @WGH- in #810
- Bump google.golang.org/protobuf from 1.24.0 to 1.33.0 by @dependabot in #807
- Implement content sniffing for HTML parsing by @WGH- in #808
- Add headers/ua to robots.txt request by @makew0rld in #838
- doc: mention that OnScraped executes after OnHTML and OnXML by @aladmit in #832
New Contributors
- @twiny made their first contribution in #525
- @kcq made their first contribution in #526
- @ysung6 made their first contribution in #537
- @lingt-xyz made their first contribution in #539
- @PalanQu made their first contribution in #531
- @dasshield made their first contribution in #549
- @shubham14bajpai made their first contribution in #563
- @wowinter13 made their first contribution in #567
- @BigManing made their first contribution in #570
- @ijomeli made their first contribution in #582
- @slashapp made their first contribution in #574
- @seversky made their first contribution in #595
- @moritamori made their first contribution in #588
- @guessi made their first contribution in #605
- @eval-exec made their first contribution in #603
- @waltton made their first contribution in #620
- @alexhraber made their first contribution in #669
- @ljbink made their first contribution in #652
- @darjun made their first contribution in #629
- @joelazar made their first contribution in #638
- @SaladinoBelisario made their first contribution in #641
- @ErikOwen made their first contribution in #691
- @cuishuang made their first contribution in #736
- @guilhem made their first contribution in #722
- @kinoute made their first contribution in #650
- @dependabot made their first contribution in #756
- @zachary-walters made their first contribution in #785
- @k4lizen made their first contribution in #786
- @Shinku-Chen made their first contribution in #794
- @ideabrian made their first contribution in #809
- @makew0rld made their first contribution in #838
- @aladmit made their first contribution in #832
Full Changelog: v2.1.0...v2.2.0
Contributors
ideabrian, guilhem, and 31 other contributors