Add support for HTTP(S) proxies to connect()

[youngkylejan]

how to make the websocket connection go through with http proxy?

[aaugustin]

Are you making connections with connect or receiving connections with serve?

[youngkylejan]

yep. for example, i want there exists function like
websockets.connect('host', http_proxy_host='proxy_host', http_proxy_port=port)

because my running environment is in the mainland of china where limits connections to lots of hosts overseas. I have to make websocket connections through my socks5 proxy which communicates with shadowsocksR server.

[aaugustin]

OK. This isn't currently implemented.

[aaugustin]

This feature request is for implemting: https://tools.ietf.org/html/rfc7231#section-4.3.6

We must take care of handling TLS properly:

• connect with or without TLS to the proxy
• connect with or without TLS to the destination

This may require handling the TLS handshake manually where asyncio currently handles it.

[aaugustin]

I have a patch that manages the four cases: TLS / not TLS between client and proxy / proxy and server.

The patch needs tests and I'm not sure how to write them.

[aaugustin]

TLS between proxy and server is very messy due to https://bugs.python.org/issue23749

Also we're likely to run into issues like aio-libs/aiosmtpd#83 until we add code to handle that.

[bird]

According to https://bugs.python.org/issue23749 the bug was fixed as of 2018-05-28. Might be worth retaking a look at? @aaugustin

[aaugustin]

I can consider that bug resolved after I drop support for all Python versions before 3.7, about five years from now.

[Yevgnen]

Any news?

[aaugustin]

No.

[gpjt]

Would supporting only client proxies to which you connect over HTTP be any easier? From the discussion above, it looks like the difficulties were all when connecting to the proxy over HTTPS.

I ask because, at least as I understand it, an proxy to which you connect over HTTP is still pretty secure, even when you're using it to ultimately connect to a site over HTTPS. The pattern is:

Client makes an HTTP connection to the proxy and sends a message like

CONNECT something.com:443 HTTP/1.1

The proxy, if it's happy to proxy the request, and is able to connect to that host on that port, replies

HTTP/1.1 200 Connection established

...and then the proxy just forwards data blindly down the pipe. The data that is sent and received can be TLS (or indeed SSH or any other TCP-based protocol).

It's worth noting that everything is reasonably secure. The only information that is sent in the clear from the client to the proxy is the port number (which is effectively in the clear for all TCP connections anyway) and the hostname, which TLS handshakes already send in the clear via the SNI field.

It's possible (or perhaps likely) that I'm missing something important here, though.

[aaugustin]

Partial support would be better than no support at all, indeed.

However, since there's a working PoC in #422 that does what you said and is merely missing error handling and tests, once it's complete for HTTP proxies, I don't think it would be very hard to also complete it for HTTP proxies.

[ghost]

For anyone still looking for this, I recommend using the websocket-client library, which has supported proxies since 2014 and works perfectly. I honestly find the fact that this issue has been open for 3 years and has a "funding needed" label laughable. Why exactly do you need funding for such a simple feature? This is just an incredibly distasteful attempt at a cash grab. It would have been far better to say something like "I'm not willing to spend time on this, but PRs are welcome".

[lgrahl]

@nyuszika7h Proxies are a pain. If you find this laughable easy, then just make a PR. PRs are always welcome. The only thing distasteful here is your comment.

[ghost]

I didn't say HTTPS proxies are trivial, but definitely not the kind of thing you need to beg for funding for. There's plenty of prior art to look at.

[lgrahl]

"The community house in this area could use some cleaning. Are you available tomorrow? Don't ask for payment though, cleaning has already been done to other community houses in the past."

[aaugustin]

It would have been far better to say something like "I'm not willing to spend time on this, but PRs are welcome".

@nyuszika7h This is basically what the last comment before yours said — glad to see that we agree on this1

[weaming]

watching this, any update?

[ghost]

I can consider that bug resolved after I drop support for all Python versions before 3.7, about five years from now.

Hello from 2023! I see this comment was made in 2018 :)
And also noticed support for Python versions older than 3.7 was dropped in version 10.0, any plans to add this soon? I love the async features of this library and wouldn't want to switch to something else.

[aaugustin]

Good job following up :-) No specific plans in this area at the time being, though.

[racinette]

@kyochikuto @weaming hey. You can check out my package: https://github.com/racinette/websockets_proxy . I subclassed the original connect and added to it the functionality by using a third party asyncio proxy connection package.

[weaming]

Nic job! Although I'm not writing Python code now. You guys can try this.

[Maxhem2]

I can consider that bug resolved after I drop support for all Python versions before 3.7, about five years from now.

Five years have passed.

[aaugustin]

Yes, that obstacle no longer exists.

[ghost]

You can check out my package: https://github.com/racinette/websockets_proxy . I subclassed the original connect and added to it the functionality by using a third party asyncio proxy connection package.

I just tested this and it's working flawlessly, really appreciate it! I love how it is only a wrapper around the original module and not a modified fork, this way you can easily keep it updated with the original module.

Keep up the good job!

[racinette]

@kyochikuto thanks man, appreciate it.

It can be used as drop-in replacement via monkey-patching, in case some other package using websockets needs proxy access. I didn't consider it for now, but I can extend the module, if the community needs it.

Also, please, do star the repo, so I can gain recognition.

[ghost]

It can be used as drop-in replacement via monkey-patching, in case some other package using websockets needs proxy access. I didn't consider it for now, but I can extend the module, if the community needs it.

Drop-in replacements are always preferred, thanks!

Also, please, do star the repo, so I can gain recognition.

Done!

[Johnserf-Seed]

For anyone still looking for this, I recommend using the websocket-client library, which has supported proxies since 2014 and works perfectly. I honestly find the fact that this issue has been open for 3 years and has a "funding needed" label laughable. Why exactly do you need funding for such a simple feature? This is just an incredibly distasteful attempt at a cash grab. It would have been far better to say something like "I'm not willing to spend time on this, but PRs are welcome".

dude, it's been 6 years.

[aaugustin]

dude, it's just a hobby, I have a real job :-)