1. Cassandra is quicker than postgre and have lower change to lose data. Cassandra doesn't have foreign keys, locking mechanism and etcs, so that it's quicker on writes.
2. Everything in cassandra is a write. Insert/update/delete is also write.
3. Setting a column to null/ deleting a column will create a tomestone; Deleting a row/primary key/partitio will create a single row tomestone
4. Could adjust tombstone_warn_threshold and tombstone_failure_threshold in cassandra.yaml.
5. Could adjust gc_grace_seconds when creating table
6. Hitting tombstone limit only happens per query.
Related Attributes
Delete will create tombstones
-
tombstone_warn_threshold: 1000 (default), could be found in cassandra.yaml
-
tombstone_failure_threshold: 100000 (default), could be found in cassandra.yaml
-
tombstone_compaction_interval: table attribute
-
min_compaction_threshold: table attribute #Compaction will only be eligible after min_compaction_threshold SSTables exist, by default it’s 4.
-
gc_grace_seconds: table attribute
-
snapshot_before_compaction: false
Check table attributes here http://docs.datastax.com/en/cassandra/2.1/cassandra/reference/referenceTableAttributes.html
Could consider using DateTieredCompactionStrategy instead of the default SizeTieredCompactionStrategy.
Cassandra MBean
Use Jconsole to remotely connect to:
hostname:7199
e.g. localhost:7199
Check/change the TombstoneFailureThreshold attribute inside StorageService MBean.
Force a flush and compaction
sudo nodetool -h localhost -p 7199 -u OC_APP_RAINBOWDBA -pw a3c224d4b89192d2ea3ea943dd7e9648 flush rainbowdba undeliveredmessage
sudo nodetool -h localhost -p 7199 -u OC_APP_RAINBOWDBA -pw a3c224d4b89192d2ea3ea943dd7e9648 compact rainbowdba undeliveredmessage
Deleted rows will only disappear when gc_grace_seconds time passed and a flush and compaction has been forced
Truncating Table
Truncating a table is an immediate operation and won’t leave any tomestones.
Don’t insert Null into columns
Inserting a null value to the column will leave a cell tomestone. Deleting a partition/row will also create a single row tombstone.
Deleting a partition will create a partition tomestone and override the existing cell tomestones. This only happens in memory table not on the disk. Not sure whether creating a partition tomestone will cause a compaction of the cell tomestones on disk.
Using TTL
insert into undeliveredmessage("id", "message","type") values('1','message','RAVEN') using ttl 5;
This query will result in 3 tomestone cells and one row tombstone.
Cassandra partition size limitation
In Cassandra, the maximum number of cells (rows x columns) in a single partition is 2 billion.
Additionally, a single column value may not be larger than 2GB. Partitions greater than 100Mb can cause significant pressure on the heap.
Performance Test
Test script TestCassandraPerformance.java could be found in
Cassandra version: 2.2.3, cqlsh 5.0.1
1. TombstoneFailureThreshold = 500
Seems persist 102000 rows and then delete them won’t hit the limit of the tomestone.
2. TombstoneFailureThreshold = 1
insert into undeliveredmessage("id","message","sent","type") values('3', 'message3', True, null);
and then select * from undeliveredmessage is fine
2. TombstoneFailureThreshold = 1
insert into undeliveredmessage("id","message","sent","type") values('3', 'message3', null, null);
and then select * from undeliveredmessage will hit the tomestone limit
|
deleted rows number |
existing rows number |
locally recovery time |
vector 2 recovery time |
|
150_000 * 9 |
Operation Timed Out |
Operation Timed Out |
|
|
150_000 * 5 |
9_072 ms |
10_152 ms |
|
|
150_000 * 3 |
5_679 ms |
7_957 ms |
|
|
150_000 * 2 |
3_025 ms |
5_218 ms |
|
|
150_000 |
1_326 ms |
1_879 ms |
|
|
150_000 |
158 ms |
333 ms |
|
|
150_000 *2 |
562 ms |
1_963 ms |
|
|
150_000 *3 |
2_223 ms |
3_833 ms |
|
|
150_000 *5 |
3_476 ms |
9_726 ms |
|
|
150_000 *10 |
Operation Timed Out |
Operation Timed Out |
|
|
150_000 |
150_000 |
1_321 ms |
3_735 ms |
|
150_000 *2 |
150_000 |
1_893 ms |
4_939 ms |
Note that we will hit timeout issue when having 150_000 *10 deleted rows in the table.
Hitting tombstone limit
For Dash you should see
com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: localhost/127.0.0.1:9042 (com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency ONE (1 responses were required but only 0 replica responded)))
at com.datastax.driver.core.ControlConnection.reconnectInternal(ControlConnection.java:223)
For query in command line, you should see something like:
Traceback (most recent call last):
File "/usr/bin/cqlsh.py", line 1172, in perform_simple_statement
rows = future.result(self.session.default_timeout)
File "/usr/share/cassandra/lib/cassandra-driver-internal-only-2.7.2.zip/cassandra-driver-2.7.2/cassandra/cluster.py", line 3347, in result
raise self._final_exception
ReadFailure: code=1300 [Replica(s) failed to execute read] message="Operation failed - received 0 responses and 1 failures" info={'failures': 1, 'received_responses': 0, 'required_responses': 1, 'consistency': 'ONE'}