Big Endianess

[guyou]

I declared the following packet:

big_endian_packets

// https://ccsds.org/Pubs/133x0b2e2.pdf
struct PacketPrimaryHeader {
  packet_version_number: 3,
  packet_type: 1,
  secondary_header: 1,
  apid: 11,
}

packet CcsdsSpacePacket {
  header: PacketPrimaryHeader,
}

Then I use it in the folowing Rust code:

    let c = ccsds::CcsdsSpacePacket {
        header: PacketPrimaryHeader {
            packet_version_number: 0,
            packet_type: 1,
            secondary_header: 0,
            apid: 1,
        },
    };
    let b = c.encode_to_vec().unwrap();
    let file = File::create("ccsds.bin").unwrap();
    let mut writer = BufWriter::new(file);
    writer.write_all(&b);
    writer.flush();

I'm surprised by the result

$ xxd -b ~/Devel/pdl/pdl-tests/ccsds.bin 
00000000: 00000000 00101000                                      .(

I expected

00000000: 00010000 00000001                                      ..

What's wrong in my declaration?

[guyou]

Note: I double checked with Kaitai and the following struct

meta:
  id: ccsds
  title: CCSDS Packet
  file-extension: bin
  endian: be
  bit-endian: be
seq:
  - id: packet_version_number
    type: b3
  - id: packet_type
    type: b1
  - id: secondary_header
    type: b1
  - id: apid
    type: b11

The producd ccsds.bin is not read as expected. If I manually create a file with 00010000 00000001 the Kaitai struct read fields correctly (packet_type=1, apid=1).

[hchataing]

The toplevel endianness selection is applied at octet level, not at the field declaration. i.e. it indicates if packed bitfields are byte swapped when encoded or not. The fields are always declared in little endian order.

Your declaration produces the following memory layout:

//     MSB                                   LSB
//     7    6    5    4    3    2    1    0
//     +---------------------------------------+
//  0  |            apid[10:3]                 |
//     +---------------------------------------+
//  1  | apid[2:0]    | sh | pt |  pvn         |
//     +---------------------------------------+
//
struct PacketPrimaryHeader {
  packet_version_number: 3,
  packet_type: 1,
  secondary_header: 1,
  apid: 11,
}

What I am assuming you want is:

//     MSB                                   LSB
//     7    6    5    4    3    2    1    0
//     +---------------------------------------+
//  0  | pvn          | pt | sh | apid[2:0]    |
//     +---------------------------------------+
//  1  |            apid[10:3]                 |
//     +---------------------------------------+
//
struct PacketPrimaryHeader {
  apid: 11,
  secondary_header: 1,
  packet_type: 1,
  packet_version_number: 3,
}

[guyou]

Thanks for the answer. I totally missed this information.
Does it make sense to add a new flag in order to control the order of fields?

[hchataing]

To be very honest everyone finds the endianness model counter intuitive.
It could make sense to also swap the field order when big endian is used 👍

We have also played with the idea of adding a compound field structure to make the field groups more explicit.
Do you think this is more readable ?

struct Test {
   field1 : 16, // round number of octets, compound declaration not needed.
   // compound field declaration with explicit endianness.
   be {
      field2 : 1
      field3 : 8
      field4 : 7
    }
}

[gbonnefille]

I'm not really familiar with endianess, I just know its existence and how to deal with it. But it's not my daily job. So, for a user like me, I just want to ignore it and simply declare it. So, I don't know if there is any logical way to describe the fields order. The standard I know use network endianess (big). In such situation, as I read my packet declaration from left to right, I just want to describe them top to bottom. Is it more natural to stack the field declaration in reverse order for little endian? This seems counter intuitive for me, but once again this is not my daily job.

Since you alert me on this "feature" I checked the code. I imagined a new flag field_order_declaration = { "little_endian_fields_order" | "big_endian_fields_order" }. Then, I imagined to adapt the code that list the fields in order to process them in a reverse order, if needed. But this seems to have a significant impact on the code.

[guyou]

I did some experiments with big endianess and sub-byte fiields.

big_endian_packets

//     MSB                                   LSB
//     7    6    5    4    3    2    1    0
//     +---------------------------------------+
//  0  |         b         |        a          |
//     +---------------------------------------+
//  1  |         d         |        c          |
//     +---------------------------------------+
//  2  |         f         |        e          |
//     +---------------------------------------+
//
packet Sequence {
  b: 4,
  a: 4,
  d: 4,
  c: 4,
  f: 4,
  e: 4,
}

//     MSB                                   LSB
//     7    6    5    4    3    2    1    0
//     +---------------------------------------+
//  0  |        a          |        b          |
//     +---------------------------------------+
//  1  |         b         |        c          |
//     +---------------------------------------+
//  2  |                   c                   |
//     +---------------------------------------+
//
packet Header {
    c: 12,
    b: 8,
    a: 4,
}

The user has to know many details of the implementation in order to know how to organize the struct declaration.
In the Header example, we have to invert ALL fields. But in Sequence example, we just have to invert the fields by two.

[hchataing]

The fields are accrued to an octet boundary, then endianness is applied to the compound. That's the estimate used to find field boundaries.

I don't think there can be a non ambiguous representation for sub-fields that does not involve an explicit compound / bitfield declaration. Can you share the specification that you are implementing ? I am curious to know how they present the field encoding / memory layout.

Using a compound be block that ensures:

1. inner fields must add up to an integral octet width
2. inner fields a1 .. aN are encoded as
   a. little endian : aN || .. || a1 written as LSB first, or maybe a1 || .. || aN to keep the same field order always
   b. big endian : a1 || .. || aN written MSB first

packet Sequence {
  be {
    a: 4,
    b: 4,
  }
  be {
    c: 4,
    d: 4,
  }
  be { 
    e: 4,
    f: 4,
  }
}

packet Header {
  be {
    a: 12,
    b: 8,
    c: 4,
  }
}

[gbonnefille]

The specification I'm trying to implement is described in https://ccsds.org/Pubs/133x0b2e2.pdf
As you can note, it is big-endian.

[gbonnefille]

I do not understand your comment about the compound be block as it does not work for me.

[hchataing]

I do not understand your comment about the compound be block as it does not work for me.

This is not yet implemented, this is something we could add to make the endianness handling less ambiguous

[gbonnefille]

Ah, OK!

If a global declaration in the spirit of big_endian_packets is not possible (think of big_endian_fields), I would suggest to introduce some attributes to packet/struct/group.

For example, Rust inspired:

#[be]
group PacketId {
    packet_type: 1,
    secondary_header_flag: 1,
    apid: 11,
}

Or Java/Python

@be
group PacketId {
    packet_type: 1,
    secondary_header_flag: 1,
    apid: 11,
}

Or :

group PacketId<be> {
    packet_type: 1,
    secondary_header_flag: 1,
    apid: 11,
}

[gbonnefille]

Can I do something to help having a more intuitive field order in bid endian context?

[hchataing]

Sorry for the delay; I have been thinking some more about your different comments.
Switching the field order in big endian packets to be big endian would definitely help resolve ambiguities about the memory layout.
I could make this change with a major version bump to make sure to break existing usages...

Coming back to your suggestion to add a packet annotation to selectively change the endianness, that sounds interesting. Not sure that there is a use case of it yet however.

[gbonnefille]

Coming back to your suggestion to add a packet annotation to selectively change the endianness, that sounds interesting. Not sure that there is a use case of it yet however.

I have to admit that, from my point of view of user, a global declaration is enough, no need to have the choice at lower level (Group, Struct, Packet).
For example:

big_endian_packets
big_endian_fields