I am trying to convert a simple URLSession request in Swift to using NWConnection. This is because I want to make the request using a Proxy that requires Authentication. I posted this SO Question about using a proxy with URLSession. Unfortunately no one answered it but I found a fix by using NWConnection instead.

My proxy works 100%, but something about my setup is not working and I get no response from the server. I'd appreciate some help converting this basic GET request to ones that utilizes NWConnection. The proxy I included works feel free to use it for testing. This specific url should return 18.5 KB of data, so around 18,000 bytes.

It would be great if someone could help because Swift needs a basic wrapper that makes requests simple using Proxies that may or may not require Auth. I'll be making a OS wrapper around NWConnection to simply its usage.

Working Request

func updateOrderStatus(completion: @escaping (Bool) -> Void) {
    let orderLink = ";

    guard let url = URL(string: orderLink) else {
        completion(true)
        return
    }

    let cookieStorage = HTTPCookieStorage.shared
    let config = URLSessionConfiguration.default
    config.httpCookieStorage = cookieStorage
    config.httpCookieAcceptPolicy = .always
    let session = URLSession(configuration: config)
    var request = URLRequest(url: url)
    request.httpMethod = "GET"

    request.setValue("text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8", forHTTPHeaderField: "Accept")
    request.setValue("none", forHTTPHeaderField: "Sec-Fetch-Site")
    request.setValue("navigate", forHTTPHeaderField: "Sec-Fetch-Mode")
    request.setValue("Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/18.0.1 Safari/605.1.15", forHTTPHeaderField: "User-Agent")
    request.setValue("en-US,en;q=0.9", forHTTPHeaderField: "Accept-Language")
    request.setValue("gzip, deflate, br", forHTTPHeaderField: "Accept-Encoding")
    request.setValue("document", forHTTPHeaderField: "Sec-Fetch-Dest")
    request.setValue("u=0, i", forHTTPHeaderField: "Priority")
    
    let task = session.dataTask(with: request) { data, response, error in
        if let error = error {
            print("Request error: \(error.localizedDescription)")
            completion(false)
            return
        }

        guard let data = data else {
            completion(false)
            print("No data received")
            return
        }

        if let body = String(data: data, encoding: .utf8) {
            completion(true)
        } else {
            completion(false)
            print("Unable to decode response body")
        }
    }

    task.resume()
}

Attempted Conversion

func updateOrderStatusProxy(completion: @escaping (Bool) -> Void) {
    let orderLink = ";

    guard let url = URL(string: orderLink) else {
        completion(true)
        return
    }
    
    let proxy = "resi.wealthproxies:8000:akzaidan:x0if46jo-country-US-session-7cz6bpzy-duration-60"
    let proxyDetails = proxy.split(separator: ":").map(String.init)
    guard proxyDetails.count == 4, let port = UInt16(proxyDetails[1]) else {
        print("Invalid proxy format")
        completion(false)
        return
    }

    let proxyEndpoint = NWEndpoint.hostPort(host: .init(proxyDetails[0]),
                                            port: NWEndpoint.Port(integerLiteral: port))
    let proxyConfig = ProxyConfiguration(httpCONNECTProxy: proxyEndpoint, tlsOptions: nil)
    proxyConfig.applyCredential(username: proxyDetails[2], password: proxyDetails[3])

    let parameters = NWParameters.tcp
    let privacyContext = NWParameters.PrivacyContext(description: "ProxyConfig")
    privacyContext.proxyConfigurations = [proxyConfig]
    parameters.setPrivacyContext(privacyContext)

    let host = url.host ?? ""
    let path = url.path.isEmpty ? "/" : url.path
    let query = url.query ?? ""
    let fullPath = query.isEmpty ? path : "\(path)?\(query)"

    let connection = NWConnection(
        to: .hostPort(
            host: .init(host),
            port: .init(integerLiteral: UInt16(url.port ?? 80))
        ),
        using: parameters
    )

    connection.stateUpdateHandler = { state in
        switch state {
        case .ready:
            print("Connected to proxy: \(proxyDetails[0])")

            let httpRequest = """
            GET \(fullPath) HTTP/1.1\r
            Host: \(host)\r
            Connection: close\r
            Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8\r
            User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/18.0.1 Safari/605.1.15\r
            Accept-Language: en-US,en;q=0.9\r
            Accept-Encoding: gzip, deflate, br\r
            Sec-Fetch-Dest: document\r
            Sec-Fetch-Mode: navigate\r
            Sec-Fetch-Site: none\r
            Priority: u=0, i\r
            \r
            """

            connection.send(content: httpRequest.data(using: .utf8), completion: .contentProcessed({ error in
                if let error = error {
                    print("Failed to send request: \(error)")
                    completion(false)
                    return
                }

                // Read data until the connection is complete
                self.readAllData(connection: connection) { finalData, readError in
                    if let readError = readError {
                        print("Failed to receive response: \(readError)")
                        completion(false)
                        return
                    }

                    guard let data = finalData else {
                        print("No data received or unable to read data.")
                        completion(false)
                        return
                    }

                    if let body = String(data: data, encoding: .utf8) {
                        print("Received \(data.count) bytes")
                        print("\n\nBody is \(body)")
                        completion(true)
                    } else {
                        print("Unable to decode response body.")
                        completion(false)
                    }
                }
            }))

        case .failed(let error):
            print("Connection failed for proxy \(proxyDetails[0]): \(error)")
            completion(false)

        case .cancelled:
            print("Connection cancelled for proxy \(proxyDetails[0])")
            completion(false)

        case .waiting(let error):
            print("Connection waiting for proxy \(proxyDetails[0]): \(error)")
            completion(false)

        default:
            break
        }
    }

    connection.start(queue: .global())
}
private func readAllData(connection: NWConnection,
                         accumulatedData: Data = Data(),
                         completion: @escaping (Data?, Error?) -> Void) {
    
    connection.receive(minimumIncompleteLength: 1, maximumLength: 65536) { data, context, isComplete, error in
        
        if let error = error {
            completion(nil, error)
            return
        }
        
        // Append newly received data to what's been accumulated so far
        let newAccumulatedData = accumulatedData + (data ?? Data())

        if isComplete {
            // If isComplete is true, the server closed the connection or ended the stream
            completion(newAccumulatedData, nil)
        } else {
            // Still more data to read, so keep calling receive
            self.readAllData(connection: connection,
                             accumulatedData: newAccumulatedData,
                             completion: completion)
        }
    }
}

Correct request string

let httpRequest = "GET \("/51913883831/orders/f3ef2745f2b06c6b410e2aa8a6135847") HTTP/1.1\r\nHost: \("shops")\r\nConnection: close\r\nAccept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8\r\nUser-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/18.0.1 Safari/605.1.15\r\nAccept-Language: en-US,en;q=0.9\r\nAccept-Encoding: gzip, deflate, br\r\nSec-Fetch-Dest: document\r\nSec-Fetch-Mode: navigate\r\nSec-Fetch-Site: none\r\nPriority: u=0, i\r\n\r\n"

I am trying to convert a simple URLSession request in Swift to using NWConnection. This is because I want to make the request using a Proxy that requires Authentication. I posted this SO Question about using a proxy with URLSession. Unfortunately no one answered it but I found a fix by using NWConnection instead.

My proxy works 100%, but something about my setup is not working and I get no response from the server. I'd appreciate some help converting this basic GET request to ones that utilizes NWConnection. The proxy I included works feel free to use it for testing. This specific url should return 18.5 KB of data, so around 18,000 bytes.

It would be great if someone could help because Swift needs a basic wrapper that makes requests simple using Proxies that may or may not require Auth. I'll be making a OS wrapper around NWConnection to simply its usage.

Working Request

func updateOrderStatus(completion: @escaping (Bool) -> Void) {
    let orderLink = "https://shops/51913883831/orders/f3ef2745f2b06c6b410e2aa8a6135847"

    guard let url = URL(string: orderLink) else {
        completion(true)
        return
    }

    let cookieStorage = HTTPCookieStorage.shared
    let config = URLSessionConfiguration.default
    config.httpCookieStorage = cookieStorage
    config.httpCookieAcceptPolicy = .always
    let session = URLSession(configuration: config)
    var request = URLRequest(url: url)
    request.httpMethod = "GET"

    request.setValue("text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8", forHTTPHeaderField: "Accept")
    request.setValue("none", forHTTPHeaderField: "Sec-Fetch-Site")
    request.setValue("navigate", forHTTPHeaderField: "Sec-Fetch-Mode")
    request.setValue("Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/18.0.1 Safari/605.1.15", forHTTPHeaderField: "User-Agent")
    request.setValue("en-US,en;q=0.9", forHTTPHeaderField: "Accept-Language")
    request.setValue("gzip, deflate, br", forHTTPHeaderField: "Accept-Encoding")
    request.setValue("document", forHTTPHeaderField: "Sec-Fetch-Dest")
    request.setValue("u=0, i", forHTTPHeaderField: "Priority")
    
    let task = session.dataTask(with: request) { data, response, error in
        if let error = error {
            print("Request error: \(error.localizedDescription)")
            completion(false)
            return
        }

        guard let data = data else {
            completion(false)
            print("No data received")
            return
        }

        if let body = String(data: data, encoding: .utf8) {
            completion(true)
        } else {
            completion(false)
            print("Unable to decode response body")
        }
    }

    task.resume()
}

Attempted Conversion

func updateOrderStatusProxy(completion: @escaping (Bool) -> Void) {
    let orderLink = "https://shops/51913883831/orders/f3ef2745f2b06c6b410e2aa8a6135847"

    guard let url = URL(string: orderLink) else {
        completion(true)
        return
    }
    
    let proxy = "resi.wealthproxies:8000:akzaidan:x0if46jo-country-US-session-7cz6bpzy-duration-60"
    let proxyDetails = proxy.split(separator: ":").map(String.init)
    guard proxyDetails.count == 4, let port = UInt16(proxyDetails[1]) else {
        print("Invalid proxy format")
        completion(false)
        return
    }

    let proxyEndpoint = NWEndpoint.hostPort(host: .init(proxyDetails[0]),
                                            port: NWEndpoint.Port(integerLiteral: port))
    let proxyConfig = ProxyConfiguration(httpCONNECTProxy: proxyEndpoint, tlsOptions: nil)
    proxyConfig.applyCredential(username: proxyDetails[2], password: proxyDetails[3])

    let parameters = NWParameters.tcp
    let privacyContext = NWParameters.PrivacyContext(description: "ProxyConfig")
    privacyContext.proxyConfigurations = [proxyConfig]
    parameters.setPrivacyContext(privacyContext)

    let host = url.host ?? ""
    let path = url.path.isEmpty ? "/" : url.path
    let query = url.query ?? ""
    let fullPath = query.isEmpty ? path : "\(path)?\(query)"

    let connection = NWConnection(
        to: .hostPort(
            host: .init(host),
            port: .init(integerLiteral: UInt16(url.port ?? 80))
        ),
        using: parameters
    )

    connection.stateUpdateHandler = { state in
        switch state {
        case .ready:
            print("Connected to proxy: \(proxyDetails[0])")

            let httpRequest = """
            GET \(fullPath) HTTP/1.1\r
            Host: \(host)\r
            Connection: close\r
            Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8\r
            User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/18.0.1 Safari/605.1.15\r
            Accept-Language: en-US,en;q=0.9\r
            Accept-Encoding: gzip, deflate, br\r
            Sec-Fetch-Dest: document\r
            Sec-Fetch-Mode: navigate\r
            Sec-Fetch-Site: none\r
            Priority: u=0, i\r
            \r
            """

            connection.send(content: httpRequest.data(using: .utf8), completion: .contentProcessed({ error in
                if let error = error {
                    print("Failed to send request: \(error)")
                    completion(false)
                    return
                }

                // Read data until the connection is complete
                self.readAllData(connection: connection) { finalData, readError in
                    if let readError = readError {
                        print("Failed to receive response: \(readError)")
                        completion(false)
                        return
                    }

                    guard let data = finalData else {
                        print("No data received or unable to read data.")
                        completion(false)
                        return
                    }

                    if let body = String(data: data, encoding: .utf8) {
                        print("Received \(data.count) bytes")
                        print("\n\nBody is \(body)")
                        completion(true)
                    } else {
                        print("Unable to decode response body.")
                        completion(false)
                    }
                }
            }))

        case .failed(let error):
            print("Connection failed for proxy \(proxyDetails[0]): \(error)")
            completion(false)

        case .cancelled:
            print("Connection cancelled for proxy \(proxyDetails[0])")
            completion(false)

        case .waiting(let error):
            print("Connection waiting for proxy \(proxyDetails[0]): \(error)")
            completion(false)

        default:
            break
        }
    }

    connection.start(queue: .global())
}
private func readAllData(connection: NWConnection,
                         accumulatedData: Data = Data(),
                         completion: @escaping (Data?, Error?) -> Void) {
    
    connection.receive(minimumIncompleteLength: 1, maximumLength: 65536) { data, context, isComplete, error in
        
        if let error = error {
            completion(nil, error)
            return
        }
        
        // Append newly received data to what's been accumulated so far
        let newAccumulatedData = accumulatedData + (data ?? Data())

        if isComplete {
            // If isComplete is true, the server closed the connection or ended the stream
            completion(newAccumulatedData, nil)
        } else {
            // Still more data to read, so keep calling receive
            self.readAllData(connection: connection,
                             accumulatedData: newAccumulatedData,
                             completion: completion)
        }
    }
}

Correct request string

let httpRequest = "GET \("/51913883831/orders/f3ef2745f2b06c6b410e2aa8a6135847") HTTP/1.1\r\nHost: \("shops")\r\nConnection: close\r\nAccept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8\r\nUser-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/18.0.1 Safari/605.1.15\r\nAccept-Language: en-US,en;q=0.9\r\nAccept-Encoding: gzip, deflate, br\r\nSec-Fetch-Dest: document\r\nSec-Fetch-Mode: navigate\r\nSec-Fetch-Site: none\r\nPriority: u=0, i\r\n\r\n"

• swift
• http
• proxy
• request
• urlsession

• You should use URLSession all the way. In order to authenticate against the proxy, you may setup the Proxy-Authorization headers up front. Better though, you respond to the proxy authorization challenge (in a delegate of URLSession), and then - possibly depending on the realm of the proxy, set the corresponding proxy credentials. Basically, this authentication scheme works the same as authentication against the server you want to talk to. The difference under the hood are specific headers for the proxy. – CouchDeveloper Commented 2 days ago
• Also, I see you are using a proxy on port 8000, indicating it's an unsecured connection. ALWAYS use HTTPS! Even within VPN. Using credentials over HTTP (not HTTPs) only pretends to have a certain security and people feel safe and good. Actually, any malicious hacker has already collected all credentials within the VPN which have been sent only once over insecure HTTP. – CouchDeveloper Commented 2 days ago

1 Answer 1

I am trying to convert a simple URLSession request in Swift to using NWConnection

I suspect this is easier to do than directly implementing the challenge-response mechanism (as CouchDeveloper suggests in the comments): NWConnection will abstract for you the low-level details of the HTTP CONNECT method used for establishing a tunnel through the proxy.

I do not see any init(tls:tcp:) call in your code: your proxy, even though it is on port 8000, almost certainly expects an encrypted connection, similar to this question.

I agree, when you configure a proxy in most applications or system settings, you often see a proxy URL that starts with http://, even if the proxy itself uses TLS.
That http:// prefix in the proxy URL does not mean the connection to the proxy is unencrypted. It does indicate that the proxy is an HTTP proxy (instead of a SOCKS proxy, for instance), and that the initial interaction with the proxy will use the HTTP protocol (specifically, the CONNECT method).

    let proxyEndpoint = NWEndpoint.hostPort(host: .init(proxyDetails[0]),
                                            port: NWEndpoint.Port(integerLiteral: port))

    let tlsOptions = NWProtocolTLS.Options()
    let parameters = NWParameters(tls: tlsOptions, tcp: .init())
    let privacyContext = NWParameters.PrivacyContext(description: "ProxyConfig")

    let proxyConfig = ProxyConfiguration(httpCONNECTProxy: proxyEndpoint, tlsOptions: tlsOptions)
    proxyConfig.applyCredential(username: proxyDetails[2], password: proxyDetails[3])
    privacyContext.proxyConfigurations = [proxyConfig]
    parameters.setPrivacyContext(privacyContext)

    let connection = NWConnection(to: proxyEndpoint, using: parameters)

Note that you need to connect to the PROXY endpoint, not the destination server: The NWConnection will handle tunneling through the proxy to the final destination (shops).

The Host header must contain the destination host, not the proxy host. And said header must include a Connection: close to signal the end of the request.

    let httpRequest = """
    GET \(fullPath) HTTP/1.1\r
    Host: \(host)\r
    Connection: close\r
    Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8\r
    User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/18.0.1 Safari/605.1.15\r
    Accept-Language: en-US,en;q=0.9\r
    Accept-Encoding: gzip, deflate, br\r
    Sec-Fetch-Dest: document\r
    Sec-Fetch-Mode: navigate\r
    Sec-Fetch-Site: none\r
    Priority: u=0, i\r
    \r\n
    """

Finally, the raw data you receive from the server includes the HTTP headers (status line, content type, ...) and the body. Do not treat the entire response as the body!

    if let responseString = String(data: data, encoding: .utf8) {
        print("Received \(data.count) bytes")
        print("Response:\n\(responseString)")
        completion(true)
    } else {
        print("Unable to decode response.")
        completion(false)
    }

Ok never mind I fixed the issue.

• First of all we can't use tcp, instead I switched to let parameters = NWParameters.tls.
• Also I made the connection use port 443 instead of port 80. After doing this I got unable to decode response body this is because I specified Accept-Encoding: gzip, deflate, br. I'm not sure how to decode these 3 encodings using swift so I just removed this entry.

So using NWParameters.tls (which is equivalent to NWParameters(tls: .init())) works, while explicitly creating TCP parameters with NWParameters(tls: .init(), tcp: .init()) does not. When you use NWParameters(tls: .init()), Network framework infers some default settings that are suitable for HTTPS, including the correct port (443).
By explicitly specifying tcp: .init(), you might have been overriding those defaults and causing a mismatch with the proxy's expectations.

You also changed the connection's port to 443: the proxy is expecting a TLS connection from the start. Even though the proxy configuration uses port 8000, the destination server (shops) uses the standard HTTPS port 443. Because NWConnection is creating a tunnel through the proxy, the NWConnection needs to be told to use port 443. The proxy is listening on 8000, but the tunnel is created to port 443 on the target server.

You removed the Accept-Encoding header to get the uncompressed HTML: a valid workaround, but less efficient. Ideally, you should handle the compressed response.