Allowed list of domains #147

indrora · 2020-12-13T22:25:23Z

I've had a lot of ratelimiting because of clients that send bogus HTTPS requests:

Dec 13 14:18:02 zaibatsutel lets-proxy[29713]: 2020-12-13T14:18:02.906-0800        info        cert_manager/manager.go:156        Get certificate        {"connection_id": "47e16fb2-0884-4fc9-ab22-b7f8e2dcc21a", "domain": "www.shop.lyuaunew.zaibatsutel.net (punycode:www.shop.lyuaunew.zaibatsutel.net)", "original_domain": "www.shop.lyuaunew.zaibatsutel.net"}
Dec 13 14:18:02 zaibatsutel lets-proxy[29713]: 2020-12-13T14:18:02.923-0800        info        cert_manager/manager.go:156        Get certificate        {"connection_id": "33bfc9e3-7f9c-4d54-a5ec-12c34892dc7f", "domain": "shop.lyuaunew.zaibatsutel.net (punycode:shop.lyuaunew.zaibatsutel.net)", "original_domain": "shop.lyuaunew.zaibatsutel.net"}
Dec 13 14:22:12 zaibatsutel lets-proxy[29713]: 2020-12-13T14:22:12.727-0800        info        cert_manager/manager.go:156        Get certificate        {"connection_id": "e4ff37c0-c5aa-483f-8fe4-af9a33f76bf0", "domain": "test.de.zaibatsutel.net (punycode:test.de.zaibatsutel.net)", "original_domain": "test.de.zaibatsutel.net"}
Dec 13 14:22:12 zaibatsutel lets-proxy[29713]: 2020-12-13T14:22:12.728-0800        info        cert_manager/manager.go:156        Get certificate        {"connection_id": "185f12d2-0d32-41bb-be56-ae44126b46b5", "domain": "test.de.zaibatsutel.net (punycode:test.de.zaibatsutel.net)", "original_domain": "test.de.zaibatsutel.net"}
Dec 13 14:22:41 zaibatsutel lets-proxy[29713]: 2020-12-13T14:22:41.611-0800        info        cert_manager/manager.go:156        Get certificate        {"connection_id": "6d752590-b0ba-4dc9-8e64-ea671bf2f4a3", "domain": "magento.paperape.zaibatsutel.net (punycode:magento.paperape.zaibatsutel.net)", "original_domain": "magento.paperape.zaibatsutel.net"}

It would be nice to limit the specific names and patterns that will receive a certificate to avoid being ratelimited.

The text was updated successfully, but these errors were encountered:

rekby · 2020-12-14T10:56:30Z

Hello.

You can do it by deny all domains in blacklist, then allow good domains in whitelist

lets-proxy2/cmd/static/default-config.toml

Lines 130 to 137 in f97f3a9

    
           # Regexp in golang syntax of blacklisted domain for issue certificate. 
        
           #This list overrided by whitelist. 
        
           BlackList = "" 
        
           # Regexp in golang syntax of whitelist domains for issue certificate. 
        
           #Whitelist need for allow part of domains, which excluded by blacklist. 
        
           # 
        
           WhiteList = ""

brandymedia · 2023-04-06T08:35:17Z

Hello.

You can do it by deny all domains in blacklist, then allow good domains in whitelist

lets-proxy2/cmd/static/default-config.toml

Lines 130 to 137 in f97f3a9

# Regexp in golang syntax of blacklisted domain for issue certificate.

#This list overrided by whitelist.

BlackList = ""

# Regexp in golang syntax of whitelist domains for issue certificate.

#Whitelist need for allow part of domains, which excluded by blacklist.

#

WhiteList = ""

Hey Timofey - can you please give us an example at how you use the Whitelist config?

For example, if I only wanted the following domains to be allowed:

domainone.com
www.domainone.com
domaintwo.com
www.domaintwo.com

Do you just use an array for example:

WhiteList = ['domainone.com', 'www.domainone.com', 'domaintwo.com', 'www.domaintwo.com']

Or, is there another way to do this.

Thanks

rekby · 2023-04-06T09:07:40Z

Hello, @brandymedia.

WhiteList/BlackList is one string with one regexp. Regexp syntax described at https://github.com/google/re2/wiki/Syntax

for your scenation you can write one of:

WhiteList = "domainone\\.com|www.domainone\\.com|domaintwo\\.com|www.domaintwo\\.com"
WhiteList = "(www\\.)?(domainone\\.com|domaintwo\\.com)"
WhiteList = "(www\\.)?(domainone|domaintwo)\\.com"

Double backslash need because toml parser use \ as backslash too.

If you will use simple dot instead of "\." - dot will match with any symbol. It will work, but will not strong correct (regexp www.test.ru will allow domains wwwbtest.ru).

brandymedia · 2024-03-05T13:29:18Z

Hey Timofeym I have tried to add a whitelist as per your example but for some reason it still allows other subdomains.

So for example, I only want to accept www and the naked domain, but not anything.domainone.com for instance.

I need to prevent multiple subdomains accessing and thus creating certificates if that makes sense.

brandymedia · 2024-03-05T13:32:26Z

For instance, I am getting lots of ones like this:

sber.gitlab.r5fg6uqvkdxvod4.auth.domainone.com.ecdsa.cer
sber.gitlab.r5fg6uqvkdxvod4.auth.domainone.com.ecdsa.key
sber.gitlab.r5fg6uqvkdxvod4.auth.domainone.com.ecdsa.json

Which eventually makes the service stop.

rekby · 2024-03-06T07:13:19Z

Ow, it may by confused by blacklist/whitelist rules. Do you have blacklist rules?

If you want use whitelist only - you have to add BlackList=".*" (deny all).

I add more explain to config and tests with PR https://github.com/rekby/lets-proxy2/pull/213/files

Is it solve your question?

brandymedia · 2024-03-06T08:06:18Z

Thanks Timofey, that make sense - I appreciate your fast response and solution.

I'm currently testing this to see if I can prevent all the spammy certificates.

Is there a way to remove the current certificates for a domain, so I can test whether the blacklist/whitelist is working?

I tried removing the files in the storage folder, but no new ones are added from that domain when I visit the website?

Do I need to remove them from somewhere else?

Thanks for your help.

rekby · 2024-03-06T08:16:26Z

I tried removing the files in the storage folder, but no new ones are added from that domain when I visit the website?

remove storage folder is ok way. Did you restart the proxy?

Do I need to remove them from somewhere else?

No, the proxy doesn't store anything outside from storage folder.

brandymedia · 2024-03-06T08:19:45Z

Yes, I did restart the service but then no new files are created for the domain I had removed in the storage... but it still seems to load fine over https which is confusing...

I'm getting hit by tons of requests from what I can only imagine is a wildcard subdomain which is trying to create certificates.

Looking at the logs, it is now 'denying' those requests.

However, I have removed the files for certain domains which should be blocked, but they still serve over https.

brandymedia · 2024-03-06T08:27:35Z

Okay, it looks like removing the files from storage, restarting has now worked on one of the domain.

It is still however letting in files like:

mail.www.domainone.com.ecdsa.cer

I've set Blackilist to:

BlackList=".*"

And whitelist to:

WhiteList = "domainone\.com|www.domainone\.com

As I only want to allow the www and naked versions. Not any other random subdomain.

Thanks

rekby · 2024-03-06T08:38:53Z

try to use marks of start/end string, because regexp "domain.com" mean string with domain contains "domain.com". But can has some prefix and ending.

WhiteList = "^domainone\.com|www.domainone\.com$"

optional subdomain you can make by:

WhiteList = "^(www\.)?domainone\.com$"

brandymedia · 2024-03-06T08:50:30Z

Thanks again.

So in the second example

WhiteList = "^(www\.)?domainone\.com$"

Does that work with multiple domain names but restrict to only www and naked domain?

So for instance:

WhiteList = "^(www\.)?(domainone\.com|domaintwo\.com)$"

Is that correct syntax?

brandymedia · 2024-03-06T08:56:56Z

So for example, I only want this to work with:

domainone.com
www.domainone.com
domaintwo.com
www.domaintwo.com

etc

rekby · 2024-03-06T09:11:27Z

yes, it must work

brandymedia · 2024-03-06T09:27:08Z

I think I had an erro in my syntax... it should be:

^(www\\.)?(domainone\\.com|ama-uk\\.com|domainnametwo\\.com)$

Hopefully that will work now.

Out of interest, is there a way to actually prevent the subdomains requesting certificates at all? Currently they request and get denied.

Thanks again.

rekby · 2024-03-06T09:37:22Z

^(www\.)?(domainone\.com|ama-uk\.com|domainnametwo\.com)$

It contains double-escaped dots, try use one slash:

^(www\.)?(domainone\.com|ama-uk\.com|domainnametwo\.com)$

Out of interest, is there a way to actually prevent the subdomains requesting certificates at all? Currently they request and get denied.

What mean "prevent the subdomains requesting certificates"?
Prevent lets-proxy to request certificates for subdomains? (it is managed by the rules)
Or prevent clients to send request with subdomains?

brandymedia · 2024-03-06T10:15:19Z

It was the one slash that was throwing the error. It seemed to work with the 2 slashes. Sorry, reg ex is not my speciality 🙈

I'll test this now and let you know the outcome.

brandymedia · 2024-03-06T12:06:28Z

Unfortunately now I can not get a certificate for a domain on the www or naked. I removed all occurrences of that domain in the storage folder and now it won't add new ones for that domain.

Do you think this could be down to limits?

brandymedia · 2024-03-07T09:21:04Z

Ho Timofey,

Still working on this and I'm getting an error for a domain I am trying to get a certificate for.

"error": "order authorization error"

This was one of the domains that was issuing lots of certificates on subdomains - I removed them all from storage and now it won't create a new one for the www or naked version.

Do you know what this error is please?

rekby · 2024-03-07T16:01:08Z

@brandymedia, try update lets-proxy up to v0.29.3, I have added authorisation details to the error.

brandymedia · 2024-03-07T16:06:52Z

@rekby cool. Can you remind me of the process?

Is it a case of removing everything and the reuploading the new binary and reconfiguring everything?

Thanks again.

rekby · 2024-03-08T19:00:13Z

You should replace binary (and restart it) only.

brandymedia · 2024-05-13T13:45:28Z

Hey @rekby. Still struggling to get the whitelist to work.

I have set in the config file:

[CheckDomains]

BlackList=".*"
WhiteList = "^(www\.)?(domainone\.com|domaintwo\.com)$"

But it's still allowing other subdomains which means the server is getting spammed on subdomains creating too many requests which then brings the service down.

I only want to accept www or blank.

Thanks for your assistance.

brandymedia · 2024-05-14T08:13:36Z

As a follow up question, does lets-proxy run before or after accessing the vhosts file in Linux Ubuntu?

Trying to find a way to stop spurious subdomains from creating hundreds/thousands of certificates.

Wondering whether by removing the default vhost which catches all requests that don't have a specific vhost would help or not..?

rekby · 2024-05-15T07:52:15Z

The rules should work fine. Similar rule checked by test.

You can write actual config and example of domain, which must be denied by rules, but allowed by real. I will check it by the test.

If it isn't help to detect problem I will need debug log for issue certificate for one of bad subdomain.

brandymedia · 2024-05-15T08:19:31Z

Thanks Timofey,

Here is an example of the config file:

[CheckDomains]

BlackList=".*"

WhiteList = "^(www\\.)?(edgarbrooks\\.co\\.uk|annandalefinancial\\.co\\.uk)$"

When I look in the storage directory there are hundred if not thousands of certificates for subdomains on these domains I do not want.

Thanks again.

rekby · 2024-05-15T10:33:49Z

Can show write one of them for check by rules by hands, by test, debug on the example, etc?

Better one of latest domain. You can grep it from a log by string "Certificate issued." or command ls -lah --sort time -r | head in the storage folder.

brandymedia · 2024-05-15T11:07:14Z

I have checked the latest files using ls -lt | less in the storage folder.

The latest one on this server is - Mar 7 08:44 yandex.avito.pay.pay.r5fg6uqvkdxvod4.www.auth.ama-uk.com.ecdsa.cer

Although on some of the servers these are getting created much quicker.

What do I need to do to test this manually please?

brandymedia · 2024-05-15T11:09:16Z

For example on another of our servers we have many records created in the same minute:

-rw------- 1 lets-proxy lets-proxy 3420 May 15 12:07 blog.eforward4.axiomfinance.co.uk.ecdsa.cer
-rw------- 1 lets-proxy lets-proxy  161 May 15 12:07 blog.eforward4.axiomfinance.co.uk.ecdsa.json
-rw------- 1 lets-proxy lets-proxy  227 May 15 12:07 blog.eforward4.axiomfinance.co.uk.ecdsa.key
-rw------- 1 lets-proxy lets-proxy 3514 May 15 12:07 ozon.ozon.cdek.od0uygcil2pwhit.vmail.imsmortgages.uk.com.ecdsa.cer
-rw------- 1 lets-proxy lets-proxy  207 May 15 12:07 ozon.ozon.cdek.od0uygcil2pwhit.vmail.imsmortgages.uk.com.ecdsa.json
-rw------- 1 lets-proxy lets-proxy  227 May 15 12:07 ozon.ozon.cdek.od0uygcil2pwhit.vmail.imsmortgages.uk.com.ecdsa.key
-rw------- 1 lets-proxy lets-proxy 3501 May 15 12:07 gitlab.gitlab.git.git.git.grelay1.imsmortgages.uk.com.ecdsa.cer
-rw------- 1 lets-proxy lets-proxy  201 May 15 12:07 gitlab.gitlab.git.git.git.grelay1.imsmortgages.uk.com.ecdsa.json
-rw------- 1 lets-proxy lets-proxy  227 May 15 12:07 gitlab.gitlab.git.git.git.grelay1.imsmortgages.uk.com.ecdsa.key
-rw------- 1 lets-proxy lets-proxy  219 May 15 12:07 git.mwgi5cibs0ekr8c.keygqkhiyj2mu9q.booking.axiomfinance.co.uk.ecdsa.json
-rw------- 1 lets-proxy lets-proxy 3542 May 15 12:07 git.mwgi5cibs0ekr8c.keygqkhiyj2mu9q.booking.axiomfinance.co.uk.ecdsa.cer
-rw------- 1 lets-proxy lets-proxy  227 May 15 12:07 git.mwgi5cibs0ekr8c.keygqkhiyj2mu9q.booking.axiomfinance.co.uk.ecdsa.key
-rw------- 1 lets-proxy lets-proxy 3757 May 15 12:07 www.www.www.yxfqoqp4ql1kntzn.axiomfinance.co.uk.rsa.cer
-rw------- 1 lets-proxy lets-proxy  189 May 15 12:07 www.www.www.yxfqoqp4ql1kntzn.axiomfinance.co.uk.rsa.json
-rw------- 1 lets-proxy lets-proxy 1679 May 15 12:07 www.www.www.yxfqoqp4ql1kntzn.axiomfinance.co.uk.rsa.key
-rw------- 1 lets-proxy lets-proxy 3542 May 15 12:07 punh3lv801obdjk.4g1i8mw2mne74d1.manage.smartestdecisions.co.uk.ecdsa.cer
-rw------- 1 lets-proxy lets-proxy  219 May 15 12:07 punh3lv801obdjk.4g1i8mw2mne74d1.manage.smartestdecisions.co.uk.ecdsa.json
-rw------- 1 lets-proxy lets-proxy  227 May 15 12:07 punh3lv801obdjk.4g1i8mw2mne74d1.manage.smartestdecisions.co.uk.ecdsa.key

rekby closed this as completed Dec 14, 2020

rekby reopened this Mar 6, 2024

rekby added the question Further information is requested label Mar 6, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Allowed list of domains #147

Allowed list of domains #147

indrora commented Dec 13, 2020 •

edited

Loading

rekby commented Dec 14, 2020

brandymedia commented Apr 6, 2023

rekby commented Apr 6, 2023

brandymedia commented Mar 5, 2024

brandymedia commented Mar 5, 2024

rekby commented Mar 6, 2024

brandymedia commented Mar 6, 2024

rekby commented Mar 6, 2024 •

edited

Loading

brandymedia commented Mar 6, 2024

brandymedia commented Mar 6, 2024

rekby commented Mar 6, 2024 •

edited

Loading

brandymedia commented Mar 6, 2024

brandymedia commented Mar 6, 2024

rekby commented Mar 6, 2024

brandymedia commented Mar 6, 2024

rekby commented Mar 6, 2024

brandymedia commented Mar 6, 2024

brandymedia commented Mar 6, 2024

brandymedia commented Mar 7, 2024

rekby commented Mar 7, 2024

brandymedia commented Mar 7, 2024

rekby commented Mar 8, 2024

brandymedia commented May 13, 2024

brandymedia commented May 14, 2024

rekby commented May 15, 2024

brandymedia commented May 15, 2024

rekby commented May 15, 2024

brandymedia commented May 15, 2024

brandymedia commented May 15, 2024

Allowed list of domains #147

Allowed list of domains #147

Comments

indrora commented Dec 13, 2020 • edited Loading

rekby commented Dec 14, 2020

brandymedia commented Apr 6, 2023

rekby commented Apr 6, 2023

brandymedia commented Mar 5, 2024

brandymedia commented Mar 5, 2024

rekby commented Mar 6, 2024

brandymedia commented Mar 6, 2024

rekby commented Mar 6, 2024 • edited Loading

brandymedia commented Mar 6, 2024

brandymedia commented Mar 6, 2024

rekby commented Mar 6, 2024 • edited Loading

brandymedia commented Mar 6, 2024

brandymedia commented Mar 6, 2024

rekby commented Mar 6, 2024

brandymedia commented Mar 6, 2024

rekby commented Mar 6, 2024

brandymedia commented Mar 6, 2024

brandymedia commented Mar 6, 2024

brandymedia commented Mar 7, 2024

rekby commented Mar 7, 2024

brandymedia commented Mar 7, 2024

rekby commented Mar 8, 2024

brandymedia commented May 13, 2024

brandymedia commented May 14, 2024

rekby commented May 15, 2024

brandymedia commented May 15, 2024

rekby commented May 15, 2024

brandymedia commented May 15, 2024

brandymedia commented May 15, 2024

indrora commented Dec 13, 2020 •

edited

Loading

rekby commented Mar 6, 2024 •

edited

Loading

rekby commented Mar 6, 2024 •

edited

Loading