Skip to content

Commit

Permalink
refactor how we make an http request with proxy
Browse files Browse the repository at this point in the history
- 1. Net::HTTP::Proxy is obsolete
	- Refactor the current method to pass proxy arguments straight to Net::HTTP.new
	- [https://apidock.com/ruby/Net/HTTP/Proxy/class](https://apidock.com/ruby/Net/HTTP/Proxy/class) <- obsolete
	- [https://apidock.com/ruby/Net/HTTP/new/class](https://apidock.com/ruby/Net/HTTP/new/class) <- new way
	- This wil also make it easier to stub
- 2. Tiktok parser seems to be making the http request from within the parser
	- We make an http request from tiktok_item.rb, get_tiktok_api_data: https://github.com/meedan/pender/blob/1cb4a2b8b9afbe49ef82407588716026d05da8b4/app/models/parser/tiktok_item.rb#L55C1-L63C8
	- Here we are NOT using the proxy, we should be using it for tiktok
- All http requests should be coming from the same place
  • Loading branch information
vasconsaurus committed Nov 7, 2024
1 parent c67e7d8 commit 6d95bdb
Show file tree
Hide file tree
Showing 2 changed files with 13 additions and 11 deletions.
21 changes: 12 additions & 9 deletions app/models/concerns/request_helper.rb
Original file line number Diff line number Diff line change
Expand Up @@ -139,25 +139,28 @@ def request_url(url, verb = 'Get')
end

def request_uri(uri, verb = 'Get')
http = Net::HTTP.new(uri.host, uri.inferred_port)
http.read_timeout = PenderConfig.get('timeout', 30).to_i
http.use_ssl = uri.scheme == 'https'.freeze
http = self.initialize_http(uri)
headers = {
'User-Agent' => self.html_options(uri)['User-Agent'],
'Accept-Language' => LANG,
}.merge(self.get_cf_credentials(uri))

request = "Net::HTTP::#{verb}".constantize.new(uri.to_s, headers)
request['Cookie'] = self.set_cookies(uri)

http.request(request)
end

def initialize_http(uri)
http = Net::HTTP.new(uri.host, uri.inferred_port)
proxy_config = self.get_proxy(uri, :hash)
if proxy_config
proxy = Net::HTTP::Proxy(proxy_config['host'], proxy_config['port'], proxy_config['user'], proxy_config['pass'])
proxy.start(uri.host, uri.inferred_port, use_ssl: uri.scheme == 'https') do |http2|
http2.request(request)
end
else
http.request(request)
http = Net::HTTP.new(uri.host, uri.inferred_port, proxy_config['host'], proxy_config['port'], proxy_config['user'], proxy_config['pass'])
end
http.read_timeout = PenderConfig.get('timeout', 30).to_i
http.use_ssl = uri.scheme == 'https'.freeze

http
end

def decode_uri(url)
Expand Down
3 changes: 1 addition & 2 deletions app/models/parser/tiktok_item.rb
Original file line number Diff line number Diff line change
Expand Up @@ -54,8 +54,7 @@ def parse_data_for_parser(doc, _original_url, _jsonld_array)

def get_tiktok_api_data(requested_url)
uri = RequestHelper.parse_url(oembed_url)
http = Net::HTTP.new(uri.host, uri.inferred_port)
http.use_ssl = uri.scheme == 'https'
http = RequestHelper.initialize_http(uri)
headers = RequestHelper.extended_headers(uri)
request = Net::HTTP::Get.new(uri.request_uri, headers)
response = http.request(request)
Expand Down

0 comments on commit 6d95bdb

Please sign in to comment.