greek64/rtps-fpga
2
1
Fork 0
Code Issues Pull Requests Releases Wiki Activity
5f7f2685c2
rtps-fpga/src/TODO.txt

730 lines
57 KiB
Plaintext
Raw Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

* https://github.com/osrf/ros_dds/issues/7 tries to determine the feasibility of an FPGA-only DDS implementation
It was suggested using a 'OpenMS430' soft-microcontroller to run DDS middleware on-top.
- Compare resource utilization and performance with this approach
* DDS FPGA Vendors
- TwinOaks Computing Inc (CoreDX)
- OpenVPX
* Implementation makes unnecessary transitions, that are ignored in later stages.
This was a design decision to simplify complexity of each stage (and probably FMAX), but increases power consumption.
* Is the Timestamp used by something else except ordering by source? If not, does it have to be "sane"?
2.2.3.16
This QoS relies on the sender and receiving applications having their clocks sufficiently synchronized. If this is not the case
and the Service can detect it, the DataReader is allowed to use the reception timestamp instead of the source timestamp in its
computation of the ‘expiration time.’
2.2.3.17
The mechanism to set the source timestamp is middleware dependent.
- Well, it has to be synchronized with every writer updating the same instance.
* Are the Builtin Endpoints part of the DDS Specification or the RTPS specification?
- Both
* ALL Data Fields are sent on each change:
https://community.rti.com/forum-topic/sending-only-fields-have-changed
* OFFERED_INCOMPATIBLE_QOS, REQUESTED_INCOMPATIBLE_QOS? (DDS-1.4, 2.2.4.1 Communication Status)
* Does a RTPS Reader subscribe and receive ALL Instances of a keyed Topic?
i.e. where are the instances differentiated?
S.29 - Topic Kind
- When a reader "subscribes" to a topic, it receives all the instances of the matched writers (May not be ALL instances of the topic in general)
* Does a RTPS reader subscribe to more than one Writer (If not using Groups)?
- Yes
* Since only positive Sequence Numbers are valid, why is the type signed?
* Store only one locator? Select the first supported multicast? Check for active ping response? Verify with source address?
* What is the purpose of "manualLivelinessCount" in "SPDPdiscoveredParticipantData"? Since the liveliness is asserted by use of "ParticipantMessageData", what is the purpose of also sending an updated "SPDPdiscoveredParticipantData"? (Duplicates is out of the question, since it is not sent with the actual liveliness assertion)
* What does that mean?:
2.2.3.19
If reliability is BEST_EFFORT then the Service is allowed to drop samples. If the reliability is
RELIABLE, the Service will block the DataWriter or discard the sample at the DataReader in order not to lose existing
samples.
- It means that if the reliability is RELIABLE, and the Reader has no more space for a sample, the Reader will not accept any more samples (hence discard, and by extension block further writes in the Writer, if the Writer has also no more space for storing samples, since the Reader is no longer accepting his sends) until the application takes samples.
* What is now the valid Parameter List length?
According to DDSI-RTPS 9.4.2.11
The length encodes the number of octets following the length to reach the ID of the next parameter (or the ID of the sentinel). Because every parameterId starts on a 4-byte boundary, the length is always a multiple of four.
According to DDS XTypes 7.4.1.2
Unlike it is stated in [RTPS] Sub Clause 9.4.2.11 “ParameterList”, the value of the parameter length is the exact length of the serialized member. It does not account for any padding bytes that may follow the serialized member. Padding bytes may be added in order to start the next parameterID at a 4 byte offset relative to the previous parameterID.
* 32-bit Timestamps? Seriously? Ever heard of Y2k38?
* Use generic package with unconstrained arrays (VHDL-2008), and assert bounds/length inside the package.
* Count repository lines
git ls-files | grep .vhd | xargs wc -l
# Without Tests
git ls-files | grep .vhd | grep -v Tests | xargs wc -l
* Count Field in Heartbeat/Acknack
The following sentence is quite self-explanatory:
"A counter that is incremented each time a new message is sent. Provides the means to detect
duplicate messages that can result from the presence of redundant communication paths."
But then, in 8.4.15.7 it says:
"So, an implementation should ensure that same logical HEARTBEATs are tagged with the same Count."
Does that mean there are cases where I have to put the same count? What is a logical HEARTBEAT?
* Should a "Keyed" Endpoint communicate with a "Non-Keyed"? (In the sense of Entity Kind)
- Well, since the Topic determines if it is keyed or keyless, and since Endpoints are matched by topics, only Endpoints of either type will be matched.
* Is the empty String a valid Topic and Type Name?
* We can determine if a Endpoint is a Reader or Writer via the Entity ID. Is it illegal to get a SEDP with incompatible source (Reader Entity ID from Publications Announcer?)
* Can we make an array of records of uncontrained strings? Than we could make an array of variable sized strings...
* Should I also check for Minor_Version >= 4?
- Yes: 8.6 Implementations of this version of the RTPS protocol should be able to process RTPS Messages not only with the same major version but possibly higher minor versions.
* If a DATA Submessage is invalid in any way, the Sequence Number is never marked as received, and thus processing of remote Endpoints could stall on corrupt Messages.
* Can a Participant unmatch an Endpoint by marking it's announcing sequence number in a GAP message?
* Is DEADLINE per-INSTANCE or per-INSTANCE-and-WRITER? (From the perspective of a reader)
- It is per-INSTANCE
* Only a sub-part of the DDS QOS are actually relevant for the RTPS. Should I remove the QoS Specifications from the RTPS Package?
* What happens if we get a sample with a source timestamp earlier than the last sample that was accessed by the DataReader when using DESTINATION ORDER BY_SOURCE_TIMESTAMP? Is the sample dropped?
* The spec does not define the serialized Key (KEY=1 DATA MESSAGE)
- fast-rtps assumes it is the Key Hash
- opendds sends a full Payload (including Payload Header) containing a KeyHolder(Type) (Option a in https://issues.omg.org/issues/DDSXTY14-38)
- opensplice does same as opendds
- CycloneDDS does same as opendds
* Assert Heartbeat period > Heartbeat Suppression Period
* Can I request (NACK) SNs that were NOT announced by the writer (> last_sn in Heartbeat)?
* Does AUTOMATIC Liveliness QoS also update the lease on write/assert_liveliness operations?
- unspecified. Only requirement is that automatic liveliness is asserted at least faster than the required period
* The Lease Duration is also updated if the Cache Change is not accepted by the DDS/HC. This in effect "skews" the "correctness" of the Writer Liveliness Protocol until the reader has no pending request from the Writer.
* If an Instance is DISPOSED, but later has no active writers, the Instance STAYS in the NOT_ALIVE_DISPOSED state.
* Is a Writer that is Disposing an Instance also Unregistering that instance? (Currently only Unregistering removes the remote Writer)
- No
* Since Lifespan is a duration, there is an inherent difference in the expiration time between writer and reader. This in addition to the fact that the reader may use the Reception time for the expiration time calculation could lead to an actual expiration duration almost double in length (If sent right before expiring locally in the writer).
* The current implementation will sent a second unregister/dispose Sample, if the user does the unregister/dispose operation a second time. Should we handle that specially?
* If a Keyed Reader receives a DATA Message with no Key hash and no Payload, it will drop it since there is no way to determine the instance (And the SN will never be accepted).
* If a Best Effort Remote Reader sends an ACKNACK, he will indirectly receive a lease deadline and may timeout (DoS Attack)
* Since the Instance Handle has to be Unique but also orderable, we could use the actual Instance Memory Base Address. Since the Instances are in a list, we also have implicitly an order to all registered Instances. [It may be necessary to add a PREV pointer to the IMF to better support the read_previous_isntance operation]
- Not possible, because according to 2.2.2.5.3.16 read_next_instance, DDS v1.4:
'This ordering is between the instance handles: It [...] must be defined even for instance handles that do not correspond to instances currently managed by the DataReader.'
* Does the DEADLINE_QOS apply also to NOT_ALIVE Instances? (Current implementation makes no distinction)
* Does TIME_BASED_FILTER also apply to meta-samples (DISPOSED, NO_WRITERS)? That is an easy way to not get convergent state in different DDS Readers. What do other implementations do?
* The Participant GUID of the ParticipantMessageData is theoretically not needed, since it is the same as the source GUID of the Packet, but tt forms the DDS key together with the kind field. Our implementation checks if it is the expected GUID and drops it otherwise.
* The Discovery Module skips a Packet it doesn't understand. The Discovery Module does not parse "serialized key" of DATA messages (since the specification does not actually define what the serialized key even is). So a SPDP DATA Packet with in-line QoS (PID_STATUS_INFO) and serialized key will be dropped if the Discovery Module does not know the GUID, and that may stall the whole process pipeline (since the SN will be never acknowledged)
* Is the parameter header of an optional member in CDR_LE also endian swapped?
- Yes. (See Figure 24, 7.4.1.2.1, DDS-XTYPES 1.3)
* Based on the examples given in 7.6.8 DDS-XTYPES v1.3 specification it seems that the string bounds do not count the NUL byte.
- Under close inspection the IDL 4.2 specification states under 7.4.1.4.4.3.2
"IDL defines the string type string consisting of a list of all possible 8-bit quantities except null."
Which means that the bound of a bounded string does not count the NUL byte.
* Currently we use one rtps_discovery_module per participant. Meaning that if we want to compile 2 seperate participants we have to actually compile 2 different systems (e.g. in seperate Libraries for testing).
It would make sense to remove this restriction, rename the rtps_discovery_module to something more generic like "discovery_module", and allow a way to set participant boundaries.
* Convert ROS MSG and SRV files to IDL files
ros2 run rosidl_adapter msg2idl.py *.msg
ros2 run rosidl_adapter srv2idl.py *.srv
ros2 run rosidl_adapter action2idl.py *.action
NOTE: According to a PR, you can also do 'rosidl translate --to idl path/to/my.action'
* The publication_handle of the DDS Specification (e.g. the one returned in the Sample Info) is an implementation specific ID that can identify local and remote DDS Entities LOCALY.
* Testbench did not catch Bug, where the Pub ACKNACK was sent to as Sub ACKNACK (ACKNACK Destinations were flipped). Extend testbench to catch this case.
* Testbench did not catch rtps_writer not sending HEARTBEATS if PUSH_MODE=FALSE & DURABILITY=TRANSIENT_LOCAL
* DATA sent before first HEARTBEAT: https://github.com/ros2/rmw_cyclonedds/issues/367
* If VHDL-2019 is used, we can use "conditional analysis statements" to see if we are in a simulation (see https://insights.sigasi.com/tech/what-is-new-in-vhdl-2019-part4/)
* [2.2.3.12 TIME_BASED_FILTER, DDS 1.4] states:
"In the case where the reliability QoS kind is RELIABLE then in steady-state, defined as the situation where the DataWriter
does not write new samples for a period “long” compared to the minimum_separation, the system should guarantee delivery
the last sample to the DataReader."
We have to change the RTPS Reader to request the last SN, if the RTPS Writer did not publish for a minimum_separation period.
* [8.4.7.1 RTPS Writer, DDSI-RTPS 2.3] states:
"nackSuppressionDuration = ignore requests for data from negative acknowledgments that arrive ‘too soon’ after the corresponding change is sent."
* According to [Table 8.9, 8.2.6 The RTPS Endpoint, DDSI-RTPS 2.3], topicKind "indicates wether the Endpoint supports instance lifecycle management operations", while at the same time "indicates wether the Endpoint is associated with a DataType that has defined some fields as containing the DDS Key".
This implies that key-less Topics DO NOT have instance lifecycle management operations (i.e. no register/unregister/dispose operations)
* Section [8.7 Implementing DDS QoS and advanced DDS features using RTPS] states:
"This sub clause forms a normative part of the specification for the purpose of interoperability."
- Which means that it is part of the Specification and NOT optional. Hence why 8.4.2.2.5 requires writers to send Writer Group Information for the purposes of interoperability.
- Since LIVELINESS is also a DDS QoS why is the Writer Liveliness Protocol described in section [8.4.13 Writer Liveliness Protocol] of the Behaviour Module? It should also be described in section 8.7. As a matter of fact, section 8.4.13 explicitly states "The DDS specification requires the presence of a liveliness mechanism. RTPS realizes this requirement with the Writer Liveliness Protocol." while section 8.7 describes how the components introduced in section 8.4.13 can be used to implement the LIVELINESS QoS.
* Section [9.3.1.2 Mapping of the EntityId_t] describes the correct mapping for Reader/Writers with/without key. How should Entities with GUIDs not following these rules be handled? Is it even illegal?
* Section [2.3.2 PIM to PSM Mapping Rules] of the DDS specification explicitly states "The reason is that DCPS targets 'C' as one of the key deployment languages"
* Section [9.6.2.2 Simple Discovery Protocol built-in Endpoints] implicitly defines the DDS keys of the built-in topics, by defining a key-only DATA message. Table 9.14 than also explicitly maps PID_ENDPOINT_GUID to the TopicBuiltinTopicData::key, SubscriptionBuiltinTopicData::key, and PublicationBuiltinTopicData::key fields.
Should the key mapping also be explicitly written in text form.
* I don't understand the use case of the PID_DIRECTED_WRITE of the RTPS specification. It was added with OMG issue (https://issues.omg.org/issues/DDSIRTP2-16), but is explained terribly.
"The serialized information denotes the GUIDs of the targeted reader(s)."
Isn't that the purpose of the Reader/Writer GUID of the DATA Submessage?
Can I have multiple PID_DIRECTED_WRITE?
* Since the DDS Reader is waiting on USER via the 'sample_info_ack' signal before continuing, a single user can stall all other USERs/Readers (DoS) of a vector entity of the DDS Reader.
* Source Port of SPDP is irrelevant, since it is BEST EFFORT and we do not reply (only Destination Port is of significance)
* According to "2.2.4.2.2 Changes in Read Communication Statuses" in the DDS 1.4 Specification it seems like it is meant to modify existing samples, instead of generating new data-less ones in order to transition the instance state.
* Fast-RTPS does not follow DDSI-RTPS Specification
- Open Github Issue
https://github.com/eProsima/Fast-RTPS/issues/1221
- Seems that Fast-RTPS is also not checking the Validity of Submessages according to Spec
* DDSI-RTPS 2.3 ISSUES
- 8.2.2 The History Cache
The 'get_change()' operation depicted in 8.3 is not documented.
- 8.2.2.4 get_seq_num_min
8.2.2.5 get_seq_num_max
This asume a history cache with duplicate-free sequence numbers, but because sequence number are
generated per writer, and a reader can be matched with mutliple writers, we can get duplicate
sequence numbers of different changes from different writers.
Ergo the functions are non-deterministic.
- 8.3.7.7 InfoDestination
'This message is sent from an RTPS Writer to an RTPS Reader to modify the GuidPrefix used to
interpret the Reader entityIds appearing in the Submessages that follow it.'
But state is changed as follows 'Receiver.destGuidPrefix = InfoDestination.guidPrefix'.
Isn't Reader -> Writer also valid? Does it have a specific direction?
- 8.2.3 The RTPS CacheChange
Add IDL Specification for CacheChange_t
- 8.3.4 The RTPS Message Receiver, Table 8.16 - Initial State of the Receiver
Port of UnicastReplyLocatorList should be initialized to Source Port.
- 8.3.4.1 Rules Followed by the Message Receiver
'Submessage and when it should be considered invalid.'
This belongs to the previous sentence.
- 8.3.7
"Contains information regarding the value of an application Date-object."
Should be Data-object
- 8.3.7.2.3 Validity
"inlineQos is invalid."
It is not specified what an invalid inlineQoS is.
- 8.3.7.4.3 Validity
gapList.Base >= gapStart
- 8.3.7.4.5 Logical Interpretation
'See section 8.7.6 for how DDS uses this feature.'
Wrong reference. 8.7.5 correct
- 8.3.7.5.5 Logical Interpretation
'These fields provide relate the CacheChanges of Writers belonging to a Writer Group.'
Remove provide
'See 8.7.6 for how DDS uses this feature.'
Wrong reference. 8.7.5 correct
- 8.3.7.10.3 Validity
'This Submessage is invalid when the following is true:
submessageLength in the Submessage header is too small'
But if InvalidateFlag is set, Length can be Zero. Since the length is unsigned, there cannot be an invalid length.
- 8.3.7.11.1
'Given the size of a SequenceNumberSet is limited to 256, an AckNack Submessage is limited to NACKing only those samples whose sequence number does not not exceed that of the first missing sample by more than 256.'
Remove one not
- 8.4.2.2.5 Sending Heartbeats and Gaps with Writer Group Information
This rules seems like a last minute addition and does not follow the format until now.
Maybe rewrite as "Writers must send Heartbeats and Gaps with Writer Group Information"?
'A Writer belonging to a Group shall send HEARTBEAT or GAP Submessages to its matched Readers even if the Reader has acknowledged all of that Writer’s samples.'
This sentence has nothing in common with the actual requirement/rule. Usually the first sentence following the actual requirement explains the requirement in more detail.
Is this sentnce do be understood in addition or instead of the actual requirement/rule?
'The exception to this rule is when the Writer has sent DATA or DATA_FRAG Submessages that contain the same information.'
Link section 8.7.6 which states how this information is sent
- 8.4.7 RTPS Writer Reference Implementation
According to 8.2.2 the History Cache (HC) is the interface between RTPS nad DDS, and can be invoked
by both RTPS and DDS Entities.
8.2.9 further states 'A DDS DataWriter, for example, passes data to its matching RTPS Writer through
the common HistoryCache.', implying that a DDS Writer adds changes directly to the HC, which is then
read by the RTPS writer and handled accordingly. This means that the DDS Writer is responsible for
assigning Sequence Numbers.
This goes against 8.4.7 (and Table 8.5), which states that the RTPS Writer is adding the Cache Changes to the HC
and is responsible for assigning Sequence Numbers.
- Well, according to the Virtual machine, the new_change() method is invoked on the RTPS Writer
- 8.4.13.4 Data Types Associated with Built-in Endpoints used by Writer Liveliness Protocol
Figure 8.26 states that GUID_t is used, but the PSM (9.6.2.1) maps GuidPrefix_t (Which also makes more sense)
- 8.7.2.2.1 DURABILITY
'While volatile and transient-local durability do not affect the RTPS protocol'
But in case of Durability TRANSIENT_LOCAL the writer has to send historical Data.
- 8.7.2.2.3 LIVELINESS
'If the Participant has any MANUAL_BY_PARTICIPANT Writers, implementations must check periodically
to see if write(), assert_liveliness(), dispose(), or unregister_instance() was called for any of
them.'
Elaborate if "any of them" does specify all Writers of the Participant, or only the Writers with
MANUAL_BY_PARTICIPANT Liveliness.
- 8.7.3.2 Indicating to a Reader that a Sample has been filtered
Text refs 8.3.7.2.2 for DataFlag, but shoudl also ref 8.7.4 for FilteredFlag
- 8.7.4 Changes in the Instance State
Wrong reference 9.6.3.4 (Correct 9.6.3.9)
- 8.7.4 Changes in the Instance State
'The DDS DataReader can determine the nature of the change by inspecting the InstanceState instance_state field in the SampleInfo that is returned on the DDS DataReader read or take call.'
'The serialized information within the inline QoS contains the new InstanceState, that is, whether the instance has been registered, unregistered, or disposed.'
The Specification uses the term "InstanceState" to mean both the instance states defined in the DDS specification (ALIVE, NOT_ALIVE_DISPOSED, NOT_ALIVE_NO_WRITERS) and the ChangeKind_t (ALIVE, ALIVE_FILTERED, NOT_ALIVE_DISPOSED, NOT_ALIVE_UNREGISTERED) defined in 8.2.1.2
This section should do a mapping between the two, or rename the "local" ones and use the respective terms uniformly.
- 8.7.7 Directed Write
Mention or link the PID_DIRECTED_WRITE?
- 8.7.9 Original Writer Info
"This service that forwards messages"
Replace "This" with "The"
"The RTPS protocol suports this forwarding of messages by including information of the original writer."
supports
- 9.2.2
Add newline to IDL definition after "OctetArray3 entityKey;"
- 9.3.1.2 Mapping of the EntityId_t
Add newline to IDL definition after "typedef octet OctetArray3[3];"
- 9.3.2.4 GroupDigest_t
Missing "EntityId_t" struct type name on the second struct IDL definition.
- 9.4.5.1.2 Flags
Clarify from where the endianness begins.
One might think it would begin after the Submessage Header, but the length is also endian dependent.
- 9.4.5.3 Data Submessage
writerSN is incorrectly shown as only 32 bits in width
- 9.4.5.3.1 Data Flags
"D=1 and K=1 is an invalid combination in this version of the protocol."
Does this invalidate the Submessage? Does 8.3.4.1 apply (Invalidate rest of Message)?
- 9.4.5.1.3 octetsToNextHeader
Similarly to "9.4.2.11" state that this is always a multiple of four.
- 9.6.2.2.2 Table 9.14
States that the builtinEndpointQos has no Default value, but according to
8.4.13.3
If the ParticipantProxy::builtinEndpointQos is included in the SPDPdiscoveredParticipantData, then the
BuiltinParticipantMessageWriter shall treat the BuiltinParticipantMessageReader as indicated by the flags. If
the ParticipantProxy::builtinEndpointQos is not included then the BuiltinParticipantMessageWriter shall treat
the BuiltinParticipantMessageReader as if it is configured with RELIABLE_RELIABILITY_QOS.
which means that the default value is 0.
* DDSI-RTPS 2.5 ISSUES
- 7.4.1 The Structure Module
Figure 7.2 (RTPS Structure Module) does not correspond with actual RTPS Strucuture Module defined in 8.2 (Figure 8.1)
- 8.2.4.2 The GUIDs of RTPS Participants
The bulletpoints list seems malformed and makes no sense.
- 8.7.10 Key Hash
Wrong reference 9.6.4.3 (correct 9.6.4.8)
- 9.3.1.2 Mapping of the EntityId_t
Malformed bits in entityKind description
- 9.4.5.7.1
Should be "The value of the GroupInfoFlag" (instead of LivelinessFlag)
- 9.6.1 ParameterId definitions in the HeaderExtension
"[...] shall either skip and the parameter or reject [...]" (And is superflous)
- 8.2.1 Overview
Figure 8.1 should contain the RTPS Group Entry (it even has its own section 8.2.6)
It should also get an Entry in the Table 8.1
- 8.5.4.4 Data Types associated with built-in Endpoints used by the Simple Endpoint Discovery Protocol
Figure 8.30 does not represent the types defined in DDS Specification
- 8.7.5 Group Ordered Access
"A DataReader attached to a Subscriber configured with access scope GROUP" should be Reader (RTPS), and not DataReader (DDS)
- This section has general incosistency in the use of the RTPS and DDS counterparts
- 9.3.2.5
IDL definition is missing the "EntityId_t" name in the struct definition
* DDS 1.4 ISSUES
- 2.2.3 Supported QoS
Partition is marked as RxO=No, but should be RxO=Yes? Or not?
- Existing Issue: https://issues.omg.org/issues/DDS15-245
- 2.2.3 Supported QoS
OWNERSHIP is marked optional as a whole, but defines a default (SHARED) if unspecified (Since it is RxO)
Define only the values other than SHARED as optional (consistent with other QOS Policies)
- 2.2.3 Supported QoS
Under DURABILITY_SERVICE is states "And three integers:", while there are 4 integeres
- How is History QoS affecting MAX_INSTANCES handling (if at all). When is an instance eligibale for
replacement.
- 2.2.2.4.2.5 register_instance
This operation may block and return TIMEOUT under the same circumstances described for the write operation
This operation may return OUT_OF_RESOURCES under the same circumstances described for the write operation
But the opration only returns InstanceHandle_t. Other vendors return HANDLE_NIL also on error and not only
"if the Service does not want to allocate any handle for that instance".
It should propably behave like the Lookup_Instance operation.
- General
Clarify semantic difference between lost and rejected samples.
- General
How handling of MAX_INSTANCES is affected by the HISTORY QoS Policy
- General
The DDS Specification does not explicitly state that the behaviour of the Register/Unregister/Dispose Operations have on non-keyed Topics
- General
The DDS Specification is not entirely clear how to handle transition between the NOT_ALIVE states. Going by the
petri-net state-flowchart transitions between the NOT_ALIVE states are not allowed, meaning that the first taken
NOT_ALIVE state stays until the instance is reborn. But since the Disposal of an Instance is of higher
Information value, we should support transitioning from NOT_ALIVE_NO_WRITERS to NOT_ALIVE_DISPOSED.
- General
An RTPS CacheChange is not equal to a DDS Sample. (e.g. an unregister CacheChange does not necessarily translate
to a DDS Sample). State the translation.
DESIGN DECISIONS
================
GENERAL
-------
* !REJECTED!
In order to save memory GUID should only be saved once. Decision was made to replace GUID with internal reference
index. Discovery module is responsible for saving the GUID and map it to a reference index, that can then be used
by other entities. Writer Endpoints may need access to the real GUID for message fields.
2 options exist:
- All Endpoints have access to the central memory where the real GUID is saved (needs Arbiter, handle starvation)
- Writer Endpoints fill the fields with the reference index as placeholder, and a separate Entity will access
the central memory and replace the actual values
The Second option was chosen (Less resources)
RTPS Handler should lookup received message GUID in central memory (The lookup should happen in parallel with the
actual message handling):
- If not stored, and message not for Discovery Module, drop message
- If in memory, replace with reference index
The central memory is accessed by 3 Entities:
- RTPS Handler (READ, GUID Lookup)
- Placeholder Handler (READ, GUID Lookup)
- Discovery Module (WRITE, GUID Save) [Need initial Lookup? RTPS Handler should have already handled it. How
does DM know if actual GUID or reference index?]
Use a 2-port RAM with an arbiter for READ operations (Give Placeholder Handler priority to prevent DoS starvation)
* Use the lowest bit of the Heartbeat/Acknack Deadline stored in the memory to differentiate
between Delay and Suppression. This reduces the resolution from 0.23 ns to 0.47 ns
(A previous version was using the unused extra flags in the stored participant data)
* Initially the RTPS/DDS Endpoints were designed as one Endpoint per Entity. This allows maximum parallel
processing and each entity having the bare minimum HW based on generics that define the properties of
each Endpoint. Nevertheless the amount of Resources needed to synthesize are quite substantial, and there
is a (low) limit of how many Endpoints can be instantiated. This limit was reached when trying to
synthesize a ROS action server, which instantiates 9 RTPS and DDS Endpoints.
Since the only real difference between the Endpoints is the Memory, we could reuse the main state
machine for all Endpoints and just instantiate different memories.
A redesign of all Endpoints was decided, in which multiple Endpoints are simulated by one Entity.
The contained Endpoints are addressed in sequential order, meaning that we lose the parallel processing,
but since the RTPS protocol is primarily used over UDP, there is no difference in performance.
Although the ports of the entity could remain single dimension (since only one Endpoint is
reading/writing at a time), we would lose the information of which Endpoint is addressed and would have
to extend the inter-communication schema to relay this information. To avoid this, and to be backwards
compatible (allow to instantiate multiple Endpoint Entities), the dimensions of the ports of the
Endpoints will be extended by the dimension of Endpoints contained with some exceptions.
These exceptions are the all RTPS Output ports, and the read, data_in, and last_word_in RTPS Handler
and DISCOVERY Module ports. This prevents wasting resources on FIFOs, but still conveys addressing
information via the empty signal.
This works, because the RTPS Handler and DISCOVERY MODULE write in a multicast fashion (meaning that all
addressed Entities become the same data).
* The above decision brings with it another challenge. Since now the input signals are unbalanced
(empty port is vector, but read, data_in, and last_word_in are not) we need a special kind of FIFO
to connect to the input ports of the Endpoints. This special FIFO is called "vector_FIFO" and contains
a FIFO for the data, and a FIFO for the write signal, that is internally converted to the empty signal
and simulates multiple FIFOs.
DISCOVERY MODULE
----------------
* Originally we stored the mask of local matching endpoints in the memory frame of the remote participant
in order to be able to send MATCH frames only to new matches, and UNMATCH frames only to previously
matched local endpoints. This decision was reverted, and we just sent MATCH frames to the currently
matched local endpoints (non depending on if they are already matched) and UNMATCH frames to the
rest of the local endpoints (non depending on if they were previously matched).
So we basically push the responsibility to the local endpoints, which have to handle these situations
accordingly. Since META traffic is not supposed to be generated as often, this should not produce
any significant overhead. As optimization, on new matched remote endpoints UNMATCH frames can be
ignored.
* The HEARTBEATs are sent out together with the liveliness assertions. This adds a 96-Byte overhead
to the output RTPS Message. This was done to prevent having to loop through the memory to find
remote participant destination more than once.
* The Publisher, Subscriber, and Message Data is written on separate RTPS Messages, even though they are
sent simultaneously. This decision was made to support as many local Endpoints as possible. We could
make a compile-time check and sent them in the same RTPS Message/UDP Packet, but the overhead is
quite small and not worth the hassle.
RTPS ENDPOINTS
--------------
* Even though the Reader does not need to keep track of received SN with respect to each Writer with
exception of the Highest/Last received (since it only keeps the SN in order and does only need to
request from the last stored SN on), the writer does need to keep track of the requested SN (and
possibly also the acknowledgements).
This could be solved by either storing the SN in a bitmap in the endpoint data, or by storing the
requester bitmap (endpoint data address) in the cache change data.
But since the writer might drop SNs in any order, the highest and lowest SN inside the cache history
is unbounded. We can thus only reference to still available SNs, and not to GAPs.
In order to accommodate for that, we could store the lowest (and possibly highest) SN of a requested
lost SN and always send ALL GAPs in that range.
* Since the RTPS Writer only gets ACKNACK Messages from the matched Readers, and these Messages are
dropped by the rtps_handler if smaller than expected, we do not need a "READ GUARD" in the RTPS
Writer.
* Because "Once Acknowledged, Always Acknowledged" the Base of an ACKNACK can only be bigger or
equal to the SN of the last ACKNACK. It is also reasonable, that the Reader will always request
ALL missing segments each time it sends an ACKNACK (i.e. does not assume once requested, always
requested until reception). This means that during the ACKNACK response delay, we can just parse
the new request bitmap and overwrite the last old one.
* In the RTPS Writer remote Volatile Reliable Readers are initialized with a SN equal to
the last written. This means that while the Reader can access the historical SNs via ACKNACK, the SNs
are not NACKed in the HC, and can therefore be removed at any time, not depending on the state of the
remote reader.
* The RTPS Writer is only sending GAPs when processing ACKNACK Requests or sending Historical DATA.
That means that if the HC pulls the 'data_available' Signal and deletes Cache Changes before the
RTPS Writer has requested them, it will just ignore the Cache Change. A GAP will only be generated
if a reliable Reader subsequently requests this Cache Change.
* If PUSH_MODE is false, packets are only written as response to ACKNACK requests (except HEARTBEATS).
This means that a Reliable RTPS Writer in PUSH_MODE = FALSE cannot communicate with Best Effort Readers.
Historical Data is also only sent out on request (But NACKed in the HC, until the remote Reader ACKs them).
* Following the vector entity change, since the message format from RTPS reader to DDS reader contains information
specific to the individual writer (Writer ID, Lifespan Deadline), we cannot write to multiple DDS endpoints at the
same time. This means that we have to temporarily store the payload and push it to each DDS Endpoint individually.
DDS ENDPOINTS
-------------
* The meta_data (sample info) of a cache change is fixed size, and a cache change may be connected to
data (payload), which may be variable in size. For this reason, we store the cache change and
payload in separate memories. The payload size may either be fixed (in which case the memory frame
sizes are adjusted according to that), or may be variable, in which case the payload is stored in
a linked list of predefined sized memory frames. The first word of a payload contains the address
of the next linked memory frame. If this is the last frame (or if the payload is static and there
are no linked frames), the address is MAX_ADDRESS.
* !REJECTED!
The last bit of this address is the "occupied" bit. This bit signifies if the memory
frame is used or free, and is used for the insert operation to find a new empty slot. This in
effect means that all frame sizes have to be a multiple of 2 (all frame addresses have to be
aligned to 2).
* If the last payload slot of a variable sized payload is not aligned with the actual end of the
Payload slot, we mark this via a bit in the sample info memory, and store the last address of the
actual payload in the last address of the payload slot.
* !REJECTED!
The History Cache (HC) is the interface between RTPS and DDS. The History Cache contains
the Sample Info and Payload memories. The HC has two input "sides", one is connected to the DDS
and one to the RTPS entity. Housing the memories inside the HC entity and abstracting the direct
memory address via opcode requests allows the memory interface to be replaced in future (e.g. AXI
Lite). Since all memory operations are handled by the same entity, this allows some state keeping
to improve memory bandwidth. More specifically the "linked list" paradigm can be extended to also
reference empty slots (and next unread slots), to allow selecting empty slots without iterating
through the whole memory. Originally the memory was to be implemented in a true dual port fashion,
and two seperate procoesses would each satisfy the requests from one input side. This would allow
concurrent RTPS and DDS requests to be handled. The write concurrency (add and remove change) does
not allow for state keeping (first empty slot address), since it is reset by the "adding" side, but
set by the "removing" side. Because of this, it was decided against concurrent input handling in
light of the fact that the history cache will be most commonly quite large in size, and iterating
through all...
* Since most of the DDS QoS need information that is directly available to the History Cache (HC),
it makes sense to integrate most of the DDS functionality directly into the HC to save up space
and performance. Further more the needed stored information for a DDS Entity is different enough
from the generic HC defined in the RTPS Specification to warrant a separate entity for both.
The DDS Entity will directly connect to the RTPS Endpoint. A separate generic HC will be
implemented, that follows the RTPS Specification.
The RTPS Endpoint will have to output multiple versions of Changes, depending on the connected
Entity, in order to facilitate this design decision.
* Since the "reading" side needs to have consistent state during it's processing, it does not make
sense to implement dual port RAMs for the History Cache.
* Because the Key Hash has to be known in order to make an ACCEPT/REJECT decision for the new
Sample, and the fact that the Key Hash is not always given and has to be manually calculated from
the payload, we need "buffer" slots to store the Sample and Payload until the decision can be
made. This temporal "buffer" is not implemented explicitly, but implicitly by having an extra
slot in the empty list of the memory. The Sample is added to the Empty List Head, and only added
to the occupied list tail if the Sample is accepted. In case an Older Sample needs to be removed
due to the QoS policy, this is done after the Sample Addition has been finalized (has been added
to the Occupied List), because the Sample Removal will modify the Empty List and the new Sample
would be lost if done prior to the finalization.
I.e. a single slot has to be available in the Sample/Payload Memory at all times. This is easy
for the Sample Memory, but more complicated for the Payload memory due to the dynamic slot nature
that it can have. It may happen that after an addition we have a "buffer" Sample Memory Slot, but
no Payload slot. In order to mitigate this we have to actively check the payload memory prior to
the addition and either delete the oldest sample or immediately reject the operation. This may lead
to multiple Sample removals (One prior the addition in order to free up a Payload "buffer", and
one after addition as a result of QoS policies).
* Due to the requirements of read_next_instance/take_next_instance of the DDS Reader, the Instances are
inserted in numerical Key Hash order into the Instance Memory. This extra sorting logic is not needed
in the DDS Writer, where Instances are inserted normally to the end of the List.
* The Specification does not really specify how handling of MAX_INSTANCES is affected by the HISTORY QoS
Policy (if at all). The current implementation ignores the HISTORY QoS and only removes stale instances
(Instances that are Unregistered and have all Samples acknowledged).
* According to the DDS Specification the resources taken from an instance can be freed locally if the
instance is Unregistered, and globally if the Instance is Disposed (And all Instance Samples have been
ACKed/READ/Taken). There are scenarios where removing a Disposed, but still registered Instance may
lead to inconsistent behavior (see REF.txt). RTI has thus decided to only delete unregistered
instances. I will copy this behavior.
* !REJECTED!
DATA WRITER: Once an instance is unregistered, it is eligible for deletion except if it is
Re-registered, or a write operation occurs on that instance. Disposal of an unregistered Instance
does not re-register the instance (State remains NOT_ALIVE) and is still eligible for deletion.
NOTE: The statement above is incorrect, as a writer wanting to dispose an Instance has to re-register
the Instance. Hence, it is re-registered (and the disposing writer is again active), the state
Instance remains however in a NOT_ALIVE state.
* The DDS Specification does not explicitly state that the behaviour of the Register/Unregister/Dispose
Operations have on non-keyed Topics. RTI basically does a NOP and does not modify the instance in
any way. I basically do a NOP for the Register Operation (in the sense that it does not modify the
instance in any kind of way), but still implemented the Unregister and Dispose Operations, meaning
that the Data Readers will receive a state change.
* The DDS Specification states that if after an Unregister Operation "the application wants to modify
(write or dispose) the instance, it has to register it again, or else use the special handle value
HANDLE_NIL". I do not have this pre-requirement. It will return BAD_PARAMETER only if the Instance
is not in the memory anymore.
* The DDS Specification is not entirely clear how to handle transition between the NOT_ALIVE states.
Going by the petri-net state-flowchart transitions between the NOT_ALIVE states are not allowed,
meaning that the first taken NOT_ALIVE state stays until the instance is reborn. But since the Disposal
of an Instance is of higher Information value, we should support transitioning from NOT_ALIVE_NO_WRITERS
to NOT_ALIVE_DISPOSED. This i.e. is done by transitioning to ALIVE and back to NOT_ALIVE_DISPOSED on the
same received sample (not_alive_generation_counter is incremented). This is also in accordance to the
protocol, since the writer that disposes the instance effectively (re-)registers the instance, meaning
that it now has a live writer again. On the other hand there is no transition from NOT_ALIVE_DISPOSED
to NOT_ALIVE_NO_WRITERS.
* Unregister/Filtered Samples of unknown Instances are dropped. On the other hand, Dispose Samples of
unknown Instances are accepted (Implicitly adding the new Instance).
NOTE: According to [2.2.2.4.2.13 dispose, DDS 1.4]
"In general, applications are made aware of the deletion by means of operations on the DataReader
objects that already knew that instance. DataReader objects that didn't know the instance will never see it."
So I think this has to be changed to also drop Dispose Samples of unknown Instances.
* The DDS Specification is not entirely clear on what happens in the DDS Reader on reception of
Meta Samples (Unregister/Dispose Samples). One simple solution would be to make Data-less Samples for
each received Meta-Sample. But since a meta-sample may not necessarily change the state to something
visible from the Sample info (e.g. Unregister on Instance with still alive Writers or Dispose from
another Writer on already disposed Instance), it does not seem practical. (In effect only the source
timestamp and publication handle of the Sample Info could be different).
Taking this into consideration, this DDS Reader implementation only generates data-less samples if
an Instance State is triggered. Filtered Cache Changes also do not produce any Samples visible to the
user (But affect internal state, like DEADLINE and LIVELINESS state).
* In BY_SOURCE_TIMESTAMP DESTINATION_ORDER_QOS we only allow NOT_ALIVE Samples (Unregister/Dispose) to
be added to the memory, if no newer sample of the same Instance exists. This decision was made,
because otherwise we would have to rewrite the entire Instance History and recalculate State and
Generation Counters. By dropping NOT_ALIVE samples that are in the past the Instance State is always
valid and does not have to be recalculated.
This is the only case in which a Dispose Sample can be missed/dropped, and may lead to different
Data Readers having different Generation Counters (depending on their reception order).
NOTE: The NOT_ALIVE Samples are always added to the end of the list, not depending on their TS. This
is the only exception in which samples TS does not follow their list order.
* The DEADLINE check times of the DDS Entities are aligned to the release of the reset, and not on the addition of
a new instance. That means that all Instances are checked at the same time, non depending on when they were added.
IDL
---
* Since all code related to encoding/decoding the DATA stream is dependent on the IDL type
specification, we have to encapsulate that code separately and link them as necessary. Two such
dynamic Entities are defined: KEY_HOLDER, and <TYPENAME>_INTERFACE.
The KEY_HOLDER Entity contains a Byte-Wide internal memory (In size equal to the maximum key size),
that can be filled with PLAIN_CDR/PL_CDR DATA Streams, and Serialized Key Streams.
The Entity allows outputting the memory contents (Key) either in a KEY_HASH format (needs
to instantiate a MD5 calculator), or in Serialized Key Format. The Entity uses the start/opcode/ack
interface for operations (similar to the RTPS/DDS Interface).
The <TYPENAME>_INTERFACE entity has all type-components linked to ports and latched in registers/memory.
In output mode the entity is able to fill the registers/memory with a PLAIN_CDR/PL_CDR Data Stream, and
in input mode the registers are filled directly from the input ports and the Entity is able to produce
a PLAIN_CDR/PL_CDR Data Stream from the registers/memory.
Due to the type-specific nature of the entities, those are not instantiated inside the DDS Endpoints,
but will be instantiated in a separate entity (Interface) and linked through port mapping with the
DDS Endpoints.
X: Due to port mapping differences between DDS Reader and Writer the <TYPENAME>_INTERFACE is split into
<TYPENAME>_READER_INTERFACE and <TYPENAME>_WRITER_INTERFACE.
* MUTABLE extensibility is currently unsupported, as the PL_CDR encoding needs to be able to dynamically
calculate the sizes of type members, which was deemed too complicated.
* Similar to the previous decision, only optional members with fixed size are supported (Since the optional
members need a parameter list header, which has a length field).
* Whereas the DDS Writer/Reader are flexible in the handling of variable sized Payloads - they write the
payload in multiple memory slots, effectively eating away from the MAX_SAMPLES limit, since there are
only MAX_SAMPLES memory slots avialable - the decoding in TYPENAME_interface requires and explicitly
enforces absolute maximums for all Type members. Those upper bounds are either explicitly defined in
the IDL Type definition file, or implicit system default upper bound defined in a package. Encoding
does enforce those limits implicitly, since it encodes the memory/registers that are physically limited
to those specified upper bounds during static generation.
ROS
---
* RTPS/DDS use a time representation of 64-bit seconds in 32Q32 fixed point format. The ROS libraries
use a 64-bit unsigned nanosecond representation, and ROS sends time (defined in
rcl_interfaces/builtin_interfaces) in 32-bit second and 32-bit nanosecond respresentation.
An internal representation of a 64-bit nanosecond counter seems like the most sensible, but conversions
between the representations are quite resource and latency heavy.
Since the ros action server directly interfaces ros services with the builtin_interfaces definition,
it was decided that the entire server works on this representation to avoid costly conversions. This in
effect mitigates the conversion problem to the instantiating entity, but a single conversion point could
be defined that can be used throughout the system.
* Because the DATA_AVAILABLE bit of the status of the DDS Reader is reset on first read, the DATA_AVAILABLE
bit can be zero even if the DDS Reader still contains unread data. For this reason the ROS entities are
showing data_available until the first empty read after the DATA_AVAILABLE bit is set.
This has the inverse effect, that even if the DDS Reader has no more data the ROS Entity is showing DATA_AVAILABLE
(since an empty read has to be done to reset the bit). This trait has to be taken into account.
The ros_action_server, for instance, has to remove a goal in terminal state if the MAX_GOALS limit
is reached, and relying solely on the DATA_AVAILABLE signal could unnecessarily trigger a goal removal.
* It was decided to not follow the ROS Action Server RCL API to the letter, as the ROS Action server is more
involved and needs to do extra stuff, that the user should not really be bothered with. Using the provided
API as is requires the user to trigger various server functionality and also requires data to be unnecessarily
moved out and into the server. (Which makes sense, considering that there should be a RMW layer above this RCL
functionality).
Therefore a different API is conceptualized for the ROS Action server, that only requires User interaction to the
parts that actually matter to the user. That means that all ROS Service interactions are now hidden inside
the ROS Action server. The user is only prompted (via high active signals) for user interaction (like accepting
a new goal or cancellation request). The user primarily interacts with the server with goal handles, which are
the memory addresses of the stored goals in the goal memory. The user can fetch currently stored/accepted goals
and modify their state. The user can also directly publish feedback for specific goals. The goal results are
stored in an array, which the user directly interacts with (It is the responsibility of the user to store the
results in the array before transitioning the goal state to the SUCCEEDED state).
* According to the ROS "specification" the ROS action server should discard stored goal result based on a
configurable timeout period. A timeout period of -1 disables goal result removal (result are kept indefinitely),
and a timeout period of 0 removes goal result immediately (after serving any pending result requests).
Since we are resource limited, the timeout "option" -1 cannot be implemented as is.
Also, since the goal and result memories are linked, we are not storing and removing goal results, but the stored
goals themselves.
The design decision was made that if the timeout period is set to infinite, the oldest goal is removed if the memory
is full and a new goal request comes in.
The 0 timeout option is ignored entirely, since it is pointless and provides no benefit at all.
BRAINSTORMING
-------------
* Add all Participant specific configuration into a generic array (maybe array of record?) and modify the
discovery module to be centric to ALL participants. That means that the Participant Memory will
contain ALL remotely matched participants (even if they are matched only be 1 local participant).
The discovery module will also need to differentiate between the local participants for replies
(Parse RTPS GUID and set local array index).
The port interface of the discovery module will not change, meaning that ALL the endpoints of all the
local participants will be flattened into one array for communication purposes (Maybe define "static"
demangle package function?).
* Since Publisher and Subscriber Groups are static, we can also generate the GroupDigests statically
and avoid having to use a HW MD5 calculator.
* An important step for later testing automation would be to simulate the hardware implementation with
existing software implementations.
Currently I do not know how the hardware/software interconnection can work in the simulation level.
Can we build a SystemC wrapper around a software implementation and simulate it together with the
VHDL implementation? (QuestaSim is allowing mixed language simulations)
- Following post (https://discourse.ros.org/t/proposal-for-ros-2-hardware-acceleration-working-group-hawg/20112/22)
gave me a different idea. Do the HLS tools simulate PS and PL side together? If so implementing the software
implementation on the PS side and communicating to the PL side would implicitly allow to simulate both together.
- Wait, I think I am over-complicating stuff. Why do I need to integrate an existing software in its entirety in
a simulation capable form? Can I not just implement a SIMULATION-to-UDP bridge similar to the current module
sitting in the PS side connecting the FPGA implementation with a software implementation?
I guess only timing relevant problems would arise (Timeouts in the sw implementation because the hw implementation
takes to long to simulate a response).
* The DDS Specification differentiates between lost and rejected samples, but does not state what the semantic
difference between them is. RTI defines lost samples as dropped samples that will not be NACKed anymore. They
even extended the SAMPLE_LOST_STATUS with reasons.
My implementation is making a distinction between rejected, lost, and dropped. A sample is Rejected if it is dropped
without ACKing, dropped if it is dropped with ACKing, and lost if the DDS Reader never received a Sequence Number
(analog to the LOST_BY_WRITER reason of RTI). RTI started mapping rejected samples on BEST_EFFORT communication as
lost (since there is no way to request them).
I think that lost samples should be only samples that we have never any knowledge about (Probably also what the Spec
assumes, since the SAMPLE_LOST_STATUS has not reason). Thus I would not do the mapping, and document that rejected
samples can be considered lost if the communication is BEST_EFFORT.
On the other hand my implementation currently drops samples without any user feedback, which is quite bad. We should
extend the communication statuses with a DROPPED_SAMPLE_STATUS. Also I think it is stupid to only state the reason
of the last dropped sample, and we should do a flag variable (similar to the status register) collecting the
cumulative reasons, since some reasons are of higher value for the user than others (like dropping a Dispose sample
instead of a Filtered sample).
REF: https://community.rti.com/kb/statuses-changes
REF: https://community.rti.com/static/documentation/connext-dds/5.2.3/doc/manuals/connext_dds/html_files/RTI_ConnextDDS_CoreLibraries_UsersManual/Content/UsersManual/Statuses_for_DataReaders.htm
* The LivelinessChangedStatus of the DDS Reader requires remote endpoints to not be deleted when they lose liveliness.
On the other hand keeping "stale" remote endpoint indefinitely may also not be a good option, so we may want to add
a second configurable deadline that removes stale endpoints. We could even use the same liveliness deadline and
differentiate between the 2 deadlines with the last bit (similar to how we do it for HEARTBEAT/ACKNACK delay and
suppression)
* If we can find a way to dynamically 'use' a package in testbenches, we could dynamically overload some predefined
constants/functions that testbenches use and be Type independent.
PROTOCOL UNCOMPLIANCE
=====================
* Partition QoS Not Supported
* Coherent Sets Not Supported
* Built-in Endpoints (of Discovery Module) is NOT the same as a normal Endpoint
-> No User access to Data
* Known but unused Submessage IDs are treated as unknown
-> No validity check
* Inline QoS validated in Endpoint
-> Cannot invalidate Rest of Message/Packet
* RESOURCE_LIMITS applies also to "empty" samples (Samples with no valid data).
* Write/Dispose/Untergister Operations do not return (TIMEOUT). I.e. the MAX_BLOCKING_TIME is not used.
* The Participant Message Data may contain additional data, and according to DDSI-RTPS 2.3 implementations must be able
to support up to 128 Bytes of additional data.
RTPS ENDPOINT
=============
* 8.2.6
topicKind Used to indicate whether the Endpoint supports instance lifecycle management operations (see 8.7.4).
Reference in New Issue View Git Blame Copy Permalink