I'm trying to connect to Amazon Keyspaces from EC2 Windows instance, and couldn't, BUT I can successfully connect to it from my local PC with absolutely same trivial console application:
using System;
using Cassandra;
using System.Net.Security;
using System.Security.Cryptography.X509Certificates;
namespace cassandraconnectiotest;
internal class Program
{
static void Main(string[] args)
{
try
{
var amazoncert = new X509Certificate2("AmazonRootCA1.der");
// I obtained AmazonRootCA1.der by executing:
// curl .pem -o AmazonRootCA1.pem
// openssl x509 -outform der -in AmazonRootCA1.pem -out AmazonRootCA1.der
var sslOptions = new SSLOptions(System.Security.Authentication.SslProtocols.Tls13, false, null) //it will NOT work with simple new SSLOptions() for some reason
// and will return Cassandra.NoHostAvailableException: All hosts tried for query failed (tried 3.12.23.190:9142: IOException 'Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host..')
.SetCertificateCollection(new X509Certificate2Collection { amazoncert })
.SetRemoteCertValidationCallback((sender, cert, chain, sslPolicyErrors) =>
{
Console.WriteLine(cert.Subject);
if (sslPolicyErrors == SslPolicyErrors.None)
{
Console.WriteLine($"SSL Certificate is valid!");
return true; // Certificate is valid
}
else
{
// Log the SSL policy errors
Console.WriteLine($"SSL Certificate Error: {sslPolicyErrors}");
// Optionally, log details about the certificate and chain
if ((sslPolicyErrors & SslPolicyErrors.RemoteCertificateChainErrors) != 0 && chain != null)
{
foreach (var chainStatus in chain.ChainStatus)
{
Console.WriteLine($"Chain Status: {chainStatus.Status} - {chainStatus.StatusInformation}");
}
}
// Return false to reject the certificate
return false;
}
});
var cluster = Cluster.Builder()
.AddContactPoints("cassandra.us-east-2.amazonaws")
.WithPort(9142)
.WithAuthProvider(new PlainTextAuthProvider("cassandra_keyspaces+1-at-NNNNhere", @"xLk35password/2ndpartofit="))
.WithSSL(sslOptions)
.Build();
var session = cluster.Connect();
}
catch (Exception ex)
{
Console.WriteLine(ex.ToString());
}
}
}
When I'm connecting from my local machine, I'm getting number of messages like this:
CN=cassandra.us-east-2.amazonaws
SSL Certificate is valid!
CN=cassandra.us-east-2.amazonaws
SSL Certificate is valid!
so I'm assuming it's negotiation phase.
But when I'm running same app from EC2 instance, I'm getting:
Cassandra.NoHostAvailableException: All hosts tried for query failed (tried 3.12.23.188:9142: AuthenticationException 'Authentication failed, see inner exception.')
at Cassandra.Connections.Control.ControlConnection.Connect(Boolean isInitializing)
at Cassandra.Connections.Control.ControlConnection.InitAsync()
at Cassandra.Tasks.TaskHelper.WaitToCompleteAsync(Task task, Int32 timeout)
at Cassandra.Cluster.Init()
at Cassandra.Cluster.ConnectAsync(String keyspace)
at Cassandra.Tasks.TaskHelper.WaitToComplete(Task task, Int32 timeout)
at Cassandra.Tasks.TaskHelper.WaitToComplete[T](Task`1 task, Int32 timeout)
at Cassandra.Cluster.Connect(String keyspace)
at Cassandra.Cluster.Connect()
at cassandraconnectiotest.Program.Main(String[] args) in D:\git\beholdtech\trivialcassandraconnectiontest\cassandraconnectiotest\Program.cs:line 56
At the same time on EC2 instance:
tracert cassandra.us-east-2.amazonaws
gives:
Tracing route to cassandra.us-east-2.amazonaws [3.12.23.165]...
(seems to be working)
C:\>nslookup cassandra.us-east-2.amazonaws
Server: ip-172-31-0-2.us-east-2pute.internal
Address: 172.31.0.2
Non-authoritative answer:
Name: cassandra.us-east-2.amazonaws
Address: 3.12.23.190
telnet cassandra.us-east-2.amazonaws 9142
gives black screen with some effect when I type.
c:\> openssl s_client -connect cassandra.us-east-2.amazonaws:9142 -msg -debug
gives
Connecting to 3.12.23.184
CONNECTED(00000140)
... dumps...
subject=CN=cassandra.us-east-2.amazonaws
issuer=C=US, O=Amazon, CN=Amazon RSA 2048 M01
---
No client certificate CA names sent
Peer signing digest: SHA256
Peer signature type: RSA-PSS
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 5565 bytes and written 408 bytes
Verification error: unable to get local issuer certificate
---
New, TLSv1.3, Cipher is TLS_AES_128_GCM_SHA256
Protocol: TLSv1.3
Server public key is 2048 bit
This TLS version forbids renegotiation.
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 20 (unable to get local issuer certificate)
openssl s_client -connect cassandra.us-east-2.amazonaws:9142 -CAfile "C:\AmazonRootCA1.der" returns:
E4130000:error:05800088:x509 certificate routines:X509_load_cert_crl_file_ex:no certificate or crl found:crypto\x509\by_file.c:264:
If I use another certificate, sf-class2-root.crt, or sf-class2-root.cer obtained via:
$client = new-object System.Net.WebClient
$client.DownloadFile(".crt","d:\sf-class2-root.crt")
Export-Certificate -Cert $cert -FilePath "c:\sf-class2-root.cer"
c:>openssl s_client -connect cassandra.us-east-2.amazonaws:9142 -CAfile "C:\NEWTEST\sf-class2-root.crt"
will give me:
Connecting to 3.12.23.176
CONNECTED(00000140)
depth=4 C=US, O=Starfield Technologies, Inc., OU=Starfield Class 2 Certification Authority
verify return:1
depth=3 C=US, ST=Arizona, L=Scottsdale, O=Starfield Technologies, Inc., CN=Starfield Services Root Certificate Authority - G2
verify return:1
depth=2 C=US, O=Amazon, CN=Amazon Root CA 1
verify return:1
depth=1 C=US, O=Amazon, CN=Amazon RSA 2048 M01
verify return:1
depth=0 CN=cassandra.us-east-2.amazonaws
verify return:1
---
Certificate chain
0 s:CN=cassandra.us-east-2.amazonaws
i:C=US, O=Amazon, CN=Amazon RSA 2048 M01
a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
v:NotBefore: Nov 8 00:00:00 2024 GMT; NotAfter: Oct 26 23:59:59 2025 GMT
1 s:C=US, O=Amazon, CN=Amazon RSA 2048 M01
i:C=US, O=Amazon, CN=Amazon Root CA 1
a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
v:NotBefore: Aug 23 22:21:28 2022 GMT; NotAfter: Aug 23 22:21:28 2030 GMT
2 s:C=US, O=Amazon, CN=Amazon Root CA 1
i:C=US, ST=Arizona, L=Scottsdale, O=Starfield Technologies, Inc., CN=Starfield Services Root Certificate Authority - G2
a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
v:NotBefore: May 25 12:00:00 2015 GMT; NotAfter: Dec 31 01:00:00 2037 GMT
3 s:C=US, ST=Arizona, L=Scottsdale, O=Starfield Technologies, Inc., CN=Starfield Services Root Certificate Authority - G2
i:C=US, O=Starfield Technologies, Inc., OU=Starfield Class 2 Certification Authority
a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
v:NotBefore: Sep 2 00:00:00 2009 GMT; NotAfter: Jun 28 17:39:16 2034 GMT
...
-----END CERTIFICATE-----
subject=CN=cassandra.us-east-2.amazonaws
issuer=C=US, O=Amazon, CN=Amazon RSA 2048 M01
---
No client certificate CA names sent
Peer signing digest: SHA256
Peer signature type: RSA-PSS
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 5565 bytes and written 408 bytes
Verification: OK
---
New, TLSv1.3, Cipher is TLS_AES_128_GCM_SHA256
Protocol: TLSv1.3
Server public key is 2048 bit
This TLS version forbids renegotiation.
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)
---
DONE
but starting application with this sf-class2-root.crt will lead to the same result: everything works from localhost and error on EC2 instance.
Can you please advice, what to do? How to add any logging? what could be the reason? My EC2 instance has static IP and all outbound connections allowed.
It was supposed to be trivial, very quick, easy task. It took me already DAYS trying to figure out... Is there another way to connect using CassndraCSharpDriver? TRIVIAL, which will just work from any computer, either local or EC2 (I can modify code during deployment as well, adopting behavior to environment).
I'm trying to connect to Amazon Keyspaces from EC2 Windows instance, and couldn't, BUT I can successfully connect to it from my local PC with absolutely same trivial console application:
using System;
using Cassandra;
using System.Net.Security;
using System.Security.Cryptography.X509Certificates;
namespace cassandraconnectiotest;
internal class Program
{
static void Main(string[] args)
{
try
{
var amazoncert = new X509Certificate2("AmazonRootCA1.der");
// I obtained AmazonRootCA1.der by executing:
// curl https://www.amazontrust/repository/AmazonRootCA1.pem -o AmazonRootCA1.pem
// openssl x509 -outform der -in AmazonRootCA1.pem -out AmazonRootCA1.der
var sslOptions = new SSLOptions(System.Security.Authentication.SslProtocols.Tls13, false, null) //it will NOT work with simple new SSLOptions() for some reason
// and will return Cassandra.NoHostAvailableException: All hosts tried for query failed (tried 3.12.23.190:9142: IOException 'Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host..')
.SetCertificateCollection(new X509Certificate2Collection { amazoncert })
.SetRemoteCertValidationCallback((sender, cert, chain, sslPolicyErrors) =>
{
Console.WriteLine(cert.Subject);
if (sslPolicyErrors == SslPolicyErrors.None)
{
Console.WriteLine($"SSL Certificate is valid!");
return true; // Certificate is valid
}
else
{
// Log the SSL policy errors
Console.WriteLine($"SSL Certificate Error: {sslPolicyErrors}");
// Optionally, log details about the certificate and chain
if ((sslPolicyErrors & SslPolicyErrors.RemoteCertificateChainErrors) != 0 && chain != null)
{
foreach (var chainStatus in chain.ChainStatus)
{
Console.WriteLine($"Chain Status: {chainStatus.Status} - {chainStatus.StatusInformation}");
}
}
// Return false to reject the certificate
return false;
}
});
var cluster = Cluster.Builder()
.AddContactPoints("cassandra.us-east-2.amazonaws")
.WithPort(9142)
.WithAuthProvider(new PlainTextAuthProvider("cassandra_keyspaces+1-at-NNNNhere", @"xLk35password/2ndpartofit="))
.WithSSL(sslOptions)
.Build();
var session = cluster.Connect();
}
catch (Exception ex)
{
Console.WriteLine(ex.ToString());
}
}
}
When I'm connecting from my local machine, I'm getting number of messages like this:
CN=cassandra.us-east-2.amazonaws
SSL Certificate is valid!
CN=cassandra.us-east-2.amazonaws
SSL Certificate is valid!
so I'm assuming it's negotiation phase.
But when I'm running same app from EC2 instance, I'm getting:
Cassandra.NoHostAvailableException: All hosts tried for query failed (tried 3.12.23.188:9142: AuthenticationException 'Authentication failed, see inner exception.')
at Cassandra.Connections.Control.ControlConnection.Connect(Boolean isInitializing)
at Cassandra.Connections.Control.ControlConnection.InitAsync()
at Cassandra.Tasks.TaskHelper.WaitToCompleteAsync(Task task, Int32 timeout)
at Cassandra.Cluster.Init()
at Cassandra.Cluster.ConnectAsync(String keyspace)
at Cassandra.Tasks.TaskHelper.WaitToComplete(Task task, Int32 timeout)
at Cassandra.Tasks.TaskHelper.WaitToComplete[T](Task`1 task, Int32 timeout)
at Cassandra.Cluster.Connect(String keyspace)
at Cassandra.Cluster.Connect()
at cassandraconnectiotest.Program.Main(String[] args) in D:\git\beholdtech\trivialcassandraconnectiontest\cassandraconnectiotest\Program.cs:line 56
At the same time on EC2 instance:
tracert cassandra.us-east-2.amazonaws
gives:
Tracing route to cassandra.us-east-2.amazonaws [3.12.23.165]...
(seems to be working)
C:\>nslookup cassandra.us-east-2.amazonaws
Server: ip-172-31-0-2.us-east-2pute.internal
Address: 172.31.0.2
Non-authoritative answer:
Name: cassandra.us-east-2.amazonaws
Address: 3.12.23.190
telnet cassandra.us-east-2.amazonaws 9142
gives black screen with some effect when I type.
c:\> openssl s_client -connect cassandra.us-east-2.amazonaws:9142 -msg -debug
gives
Connecting to 3.12.23.184
CONNECTED(00000140)
... dumps...
subject=CN=cassandra.us-east-2.amazonaws
issuer=C=US, O=Amazon, CN=Amazon RSA 2048 M01
---
No client certificate CA names sent
Peer signing digest: SHA256
Peer signature type: RSA-PSS
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 5565 bytes and written 408 bytes
Verification error: unable to get local issuer certificate
---
New, TLSv1.3, Cipher is TLS_AES_128_GCM_SHA256
Protocol: TLSv1.3
Server public key is 2048 bit
This TLS version forbids renegotiation.
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 20 (unable to get local issuer certificate)
openssl s_client -connect cassandra.us-east-2.amazonaws:9142 -CAfile "C:\AmazonRootCA1.der" returns:
E4130000:error:05800088:x509 certificate routines:X509_load_cert_crl_file_ex:no certificate or crl found:crypto\x509\by_file.c:264:
If I use another certificate, sf-class2-root.crt, or sf-class2-root.cer obtained via:
$client = new-object System.Net.WebClient
$client.DownloadFile("https://certs.secureserver/repository/sf-class2-root.crt","d:\sf-class2-root.crt")
Export-Certificate -Cert $cert -FilePath "c:\sf-class2-root.cer"
c:>openssl s_client -connect cassandra.us-east-2.amazonaws:9142 -CAfile "C:\NEWTEST\sf-class2-root.crt"
will give me:
Connecting to 3.12.23.176
CONNECTED(00000140)
depth=4 C=US, O=Starfield Technologies, Inc., OU=Starfield Class 2 Certification Authority
verify return:1
depth=3 C=US, ST=Arizona, L=Scottsdale, O=Starfield Technologies, Inc., CN=Starfield Services Root Certificate Authority - G2
verify return:1
depth=2 C=US, O=Amazon, CN=Amazon Root CA 1
verify return:1
depth=1 C=US, O=Amazon, CN=Amazon RSA 2048 M01
verify return:1
depth=0 CN=cassandra.us-east-2.amazonaws
verify return:1
---
Certificate chain
0 s:CN=cassandra.us-east-2.amazonaws
i:C=US, O=Amazon, CN=Amazon RSA 2048 M01
a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
v:NotBefore: Nov 8 00:00:00 2024 GMT; NotAfter: Oct 26 23:59:59 2025 GMT
1 s:C=US, O=Amazon, CN=Amazon RSA 2048 M01
i:C=US, O=Amazon, CN=Amazon Root CA 1
a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
v:NotBefore: Aug 23 22:21:28 2022 GMT; NotAfter: Aug 23 22:21:28 2030 GMT
2 s:C=US, O=Amazon, CN=Amazon Root CA 1
i:C=US, ST=Arizona, L=Scottsdale, O=Starfield Technologies, Inc., CN=Starfield Services Root Certificate Authority - G2
a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
v:NotBefore: May 25 12:00:00 2015 GMT; NotAfter: Dec 31 01:00:00 2037 GMT
3 s:C=US, ST=Arizona, L=Scottsdale, O=Starfield Technologies, Inc., CN=Starfield Services Root Certificate Authority - G2
i:C=US, O=Starfield Technologies, Inc., OU=Starfield Class 2 Certification Authority
a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
v:NotBefore: Sep 2 00:00:00 2009 GMT; NotAfter: Jun 28 17:39:16 2034 GMT
...
-----END CERTIFICATE-----
subject=CN=cassandra.us-east-2.amazonaws
issuer=C=US, O=Amazon, CN=Amazon RSA 2048 M01
---
No client certificate CA names sent
Peer signing digest: SHA256
Peer signature type: RSA-PSS
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 5565 bytes and written 408 bytes
Verification: OK
---
New, TLSv1.3, Cipher is TLS_AES_128_GCM_SHA256
Protocol: TLSv1.3
Server public key is 2048 bit
This TLS version forbids renegotiation.
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)
---
DONE
but starting application with this sf-class2-root.crt will lead to the same result: everything works from localhost and error on EC2 instance.
Can you please advice, what to do? How to add any logging? what could be the reason? My EC2 instance has static IP and all outbound connections allowed.
It was supposed to be trivial, very quick, easy task. It took me already DAYS trying to figure out... Is there another way to connect using CassndraCSharpDriver? TRIVIAL, which will just work from any computer, either local or EC2 (I can modify code during deployment as well, adopting behavior to environment).
Share Improve this question edited Feb 17 at 22:22 Erick Ramirez 16.3k2 gold badges21 silver badges31 bronze badges asked Feb 17 at 20:07 Pasha MaltsevPasha Maltsev 592 silver badges10 bronze badges2 Answers
Reset to default 0When using X509Store
certificate store, you need to pass the X509Certificate2Collection
object to SSLOptions()
.
For example, to retrieve the collection of certificates:
X509Certificate2Collection collection = new X509Certificate2Collection();
X509Certificate2 amazoncert = new X509Certificate2(@"" + path + "/AmazonRootCA1.pem");
collection.Add(amazoncert);
Then you would pass the collection like this:
var cluster = Cluster.Builder()
...
.WithSSL(new SSLOptions().SetCertificateCollection(collection))
.Build();
The official Amazon Keyspaces code examples repository has a C# project that contains the full working code you can use as reference. Cheers!
Some recommendations from repost.aws and my responses... So far bad luck :(
1. Check your EC2 security group settings: Ensure that your EC2 instance's security group allows outbound traffic on port 9142 to the Amazon Keyspaces endpoint. Even though you mentioned all outbound connections are allowed, it's worth double-checking this specific port.
I have single security group with following Outbound rules:
IPv4 Type: All traffic, Protocol: All, Port range: All, Destination: 0.0.0.0/0
Check passed - outbound connections are allowed
also, I've tried
telnet cassandra.us-east-2.amazonaws 9142
it shows black screen, but they say it's OK and it actually means connection could be obtained using this port
2. Verify VPC settings: If your EC2 instance is in a VPC, make sure the VPC has the necessary route table entries to reach the internet or the Amazon Keyspaces endpoint. 3. Check Network ACLs: If you're using Network ACLs in your VPC, ensure they allow outbound traffic to the Keyspaces endpoint on port 9142. 4. DNS resolution: Confirm that your EC2 instance can resolve the Amazon Keyspaces endpoint. Your nslookup test suggests this is working, but double-check DNS settings in your VPC. 5. TLS/SSL configuration: Your OpenSSL test with the Starfield root certificate shows a successful connection. Ensure your C# application is using the correct certificate (sf-class2-root.crt) instead of AmazonRootCA1.der.
Again,
openssl s_client -connect cassandra.us-east-2.amazonaws:9142 -CAfile "C:\NEWTEST\sf-class2-root.crt"
gives
Connecting to 3.12.23.182
CONNECTED(00000140)
which means all of the above checks are passed, I'm 100% sure I'm using same certificate in C# app because it's just connects locally!
at the same time now I think issue is somewhere with network access... and it drives me crazy, since it seems openssl s_client -connect + nslookup + telnet + tracert gives good results??
6. Add more logging to your C# application: Implement more detailed exception handling and logging in your C# code. Capture and log the inner exception details, which might provide more insight into the authentication failure.
I couldn't get any more information, inner exception is null, I use global try/catch with exception.ToString()... and it seems there are no way I can go deeper with verbose tracing of CassandraCSharpDriver operations...
7. Check IAM permissions: Ensure that the IAM role associated with your EC2 instance (if any) has the necessary permissions to access Amazon Keyspaces.
IAM user has AmazonKeyspacesFullAccess policy assigned. And again, everything fails just on cluster.Connect(); step and works locally... , so this is not the case I think.
I've tried Sigv4 approach with access-key-id and secret-access-key pair. Same result - it works locally, but does NOT from EC2 windows instance.
8. Try using the AWS CLI: As a test, try connecting to Amazon Keyspaces using the AWS CLI from your EC2 instance. This can help determine if the issue is specific to your C# application or more general.
I've installed AWS CLI, configured it using "aws configure" with credentials I used for Sigv4 (access-key-id/secret-access-key pair) and executed the following commands:
aws keyspaces list-keyspaces --region us-east-2
gives me a list of my keyspaces I created!!!, no problems - everything works just great
9. Check for any proxy settings: Ensure there are no proxy settings on your EC2 instance that might interfere with the connection.
not sure how to do this...
10. Verify service-specific credentials: Double-check that the service-specific credentials you're using are correct and have not expired.
Since they work locally and with AWS CLI I don't think it's an issue...
So, as of now, I think it may be something very tricky with network connection, since CassandraCSharpDriver uses TLS and 9142 port, while AWS CLI uses HTTPS and 443 port... Porbably I have to try CQLSH which uses same 9142/TLS ( https://docs.aws.amazon/keyspaces/latest/devguide/programmatic.endpoints.html ), will do tomorrow, it requires python installation...