Mark G.

Field length max size

Recommended Posts

Paul, just to verify, when you wrote here "field" length here did you mean "record" length?

Or was that decided to be a "field" length limit, so a record with multiple fields could be longer (e.g. a chiral record would have max line length of 16+1+255+1+255) ?

Share this post


Link to post
Share on other sites
39 minutes ago, Yaron [LaserOp] said:

Paul, just to verify, when you wrote here "field" length here did you mean "record" length?

Or was that decided to be a "field" length limit, so a record with multiple fields could be longer (e.g. a chiral record would have max line length of 16+1+255+1+255) ?

Good point.  I had originally referred to what turned out to be an imaginary "record value" in the pre-distribution draft.  I was told the correct term is actually field value but you're right.  Our goal is to limit the overall record length so we cannot set a limit on the "field value" itself.  We need a term that refers to everything after the label separator.  I'm proposing "record value" which is what I've used for years in my own documentation and code.  I was actually surprised it wasn't an official term when Robert pointed it out.

In any event, I've sent that proposal to Robert so I'll work it into the draft.  We're still working on one or two other items anyway and this is an easy clarification.

 

Share this post


Link to post
Share on other sites

So, A couple of things:

1. What is the length of:

Yes, I also assumed there would be something like "record value", but it's indeed not defined. It wasn't needed previously, since the length limit of 80 was for the entire record (line). Which made sense, it had a reason (being able to see full line on text/terminal screen, or matching printers). The new limit is... arbitrary, in both intent and size limit.

While I really think the length limit should have just been waived, that isn't what happened, so need to do the best with what we have. In this case, then, I think a direct extension of the length from as it was before (still full record length) will be easier/simpler/straight-forward to modify to instead. Just change "80" to "255" for "record length" (5.1.8), and that's that. Or possibly also still extend text field data, as is already in the draft. Either way it removes the need to add a definition of "record value", since the exact same logic/semantics apply, only the number changed.

2. Length increase is a breaking change:

Something additional just occurred to me, which may complicate things more than intended. Increased length is inherently incompatible with anything that expects a limited length. Which means:

  1. While it seems silly, this may require a major version change (to 4.00) rather than a minor one (to 3.13). As the OMAV definition states "A change to the major identifier shall occur when changes to the interface occur which could affect the ability of hosts and devices to communicate. Changes to the minor version identifier shall occur ... should not affect the ability of hosts and devices to communicate".  If a machine, or host, will receive an already known/supported label, with a longer record length, without expecting it, they might (I'd like to assume realistically won't, but technically they're allowed to) not be able to process it. (e.g. labels like R or A or D records, where a series of multiple labels are there as a block to compensate for record length limit).
  2. This also means that the record length of 80 has to be maintained in cases where the supported OMAV is unknown. For example, during initialization, when devices send D records to hosts, they don't know if the host will support longer lengths.
    It also means that if machines don't send OMAV, then hosts should still always adhere to the shorter length.
  3. Actually, come to think of it, during initialization machines can (should, I'd think) send OMAV, but the host doesn't send its own OMAV back, so machines can never rely on it. Maybe it's also a good idea to have the host send OMAV back during initialization, to the minimum of it's own version and what it received from the machine?  A bit of a different topic, but seems extra relevant with this.
    Personally I'd change OMAV during initialization to "must", for both sides, but that may be overkill for now.

Share this post


Link to post
Share on other sites

I was looking at the max size from changing 80 to 255.  There are already fields that go past the 80 character limit so this helps keep us to the standard. 

Share this post


Link to post
Share on other sites

Mark,

I'm not sure what you're referring to? Currently (pre 3.13) the only labels without the record length limit of 80 characters are ZZ, WW, PP. The records for all other labels, when using ASCII format, have the 80 characters limit. (Now to either have the max record length change to 255, or to have the max record length removed and replaced by having the max record value length changed to 255, a decision that is apparently still open)

There are (again, pre 3.13) no fields that go without that limit, that I noticed. Text fields are explicitly limited to max length of 80, and the other types of fields are shorter. What fields are you thinking of that already go (rather than "could potentially go if it was allowed in the standard") beyond 80 characters?

Paul,

Which raises another few smaller points:

  1. There are multiple other places in the document which mention that the records do/don't adhere to the 80 characters record limit. Of course if that limit is increased, all of these need to change. Either to the new value instead of 80, or just to mention the record length limit without the number (which should be fine, since the record length is already defined elsewhere).
  2. For the labels that don't have to follow this limit, and therefore have a non-limited length, it would probably be clearer to explicitly clarify it means that they don't have a length limit, instead of just mentioning that they don't need to match the record length limit. It's not technically necessary, but could avoid some potential confusion.
  3. The max length of the text field, and the max length of the record (or of record value), are two separate things. Seems that the intent is to change them together, from the same previous 80 to the same new 255, but these are two different max length changes of two different, and only partially related, things. Personally I do think/agree the max length should be increased on both, but these are still two changes, not one change.

Share this post


Link to post
Share on other sites

Binary data is not limited to 80 chars.  In any event, this is really dragging out for what should be a very simple change.  We are simply trying to extend the max line length because some implementations are already doing so by necessity, which I believe is Mark's point.  The committee discussed this at two different meetings and voted on it.  Until now no one has said they felt this would be a protocol breaking change that would require a major revision.  Because the scope of the change was already discussed and voted on during the meeting, we will be moving forward with the new line lengths and limits as part of 3.13.  The new line length will be the sum of 16 characters for the record label, 1 character for the label separator, and 255 characters for whatever is on the right side of the label separator (no matter what we end up calling it).  Naturally, any other references that refer to maximum lengths of normal data (i.e. not binary data or some other data which already has an exception) will also be updated as necessary.

Share this post


Link to post
Share on other sites

Binary data was already not limited, so not a part of this discussion, as it's not talking about it.

Paul, the issue with compatibility is very simple: Say I'm now updating my software, on my company's machine, to support 3.13. I now need to send REQ=INI to a VCA Host. I have more than 80 characters of labels to send in the D records (already the case in some labs). What do I do in the REQ=INI packet?

  • If I send more than 80 characters, and the Host doesn't support 3.13, it's entirely fine for it to consider what I sent it to be an error. It could fail with an error, or ignore some of the labels sent, and possible cut one in the middle and assume I mean a different label. And this is for an existing label, not a new one for 3.13 that it would be otherwise justified to ignore. Clearly sending more than 80 is, in this case, a breaking change. Both sides are technically in 100% compliance to the standard, doing exactly what they're supposed to and allowed to do, and things (that were already working) break after my (bug free and correctly working) software update.
  • Do I not send over 80, even though I'm technically allowed to? If so, again, the DCS has to be explicitly clear that the 80 character limit has to be maintained whenever talking to machine/host that isn't known to be supporting of 3.13. That requires a change to the document which is bigger than just the number. And, again, this is essentially saying in the standard that the change is breaking and so backward compatibility has to be maintained, which means that, again, it contradicts the requirement for minor version. This one is doable and workable, and possibly much simpler to pass than making this a major version change, but it's "ugly".

I completely agree that the max length limit needs to extend/go. But you're maintaining a standard, and this is how this standard states that it handles breaking changes, and this is a breaking change, so it has to be addressed.

Share this post


Link to post
Share on other sites

It's too small of a change to trigger a major revision in my opinion but this isn't my decision, it's the committee's decision upon which they have already voted.  This was not pegged for 4.0, which we have already started discussing.  I don't really have anything further to say on the subject myself as I'm just moving forward with the committee's decision.  You are free to vote against approval and present your case for why this deserves to be delayed until 4.0 to the committee when the draft is sent out for voting.

Share this post


Link to post
Share on other sites

Did the committee discuss how much of a breaking change this is, and decided it doesn't merit a major version change? Or is it just that nobody noticed it's breaking, so the topic never came up? It certainly didn't occur to me in the previous talks, until just now (when I thought I could change the length for D records, and then it hit me that I actually shouldn't since it's breaking). It certainly seems like small and so not a breaking change, before noticing how it would be, I'm guilty of that myself.

The case is very simple. It's a breaking change (again, see my example of sending valid and fully compliant INF=REQ packet, that a valid and fully compliant host will suddenly not be able to process correctly). And the standard is explicit about breaking changes.

If this is inserted without officially considering it a breaking change, then, as I wrote, at the very least there should be a notice that the record length is still 80 for all cases where the supported version of the receiver is not known. If it's not there, that is surely not because the committee decided it shouldn't be, but because it was missed as an oversight.

In any case I certainly agree it's not all on you. Anyone else reading this has an opinion?

Share this post


Link to post
Share on other sites

Whether or not it was a breaking change was discussed.  I would suggest you listen to the meeting recordings for the last two meetings.  That will give you all the context you need.

Share this post


Link to post
Share on other sites

That's almost 8 hours, mostly not relevant for any one individual topic (like this), with no index or rough timestamps of what was discussed when.

I really can justify a few question on a forum, to people who were involved, compared to burning an entire workday...

So, back to my question, if I now change my software to support 3.13, can I use the new limit of 255 characters for D labels I send on REQ=INF packets?

Was the conclusion in the meeting indicating that I should, and if my system stops working in the lab because the LMS can't initialize, that's all fine? Or was it indicating that I shouldn't, but that there also shouldn't be any notice in DCS document letting me know that?

Yes, I'm sort of being snarky here, but this is also an entirely honest question, which I do need an answer for, because right now I do limit record length to 80 when sending D records on initialization.

Share this post


Link to post
Share on other sites
1 minute ago, Yaron [LaserOp] said:

That's almost 8 hours, mostly not relevant for any one individual topic (like this), with no index or rough timestamps of what was discussed when.

 

You have also posted questions on the engraving reference system which is the vast majority of the discussion of both meetings.  Most of the easier topics are covered within the first hour or two of each meeting because we save the best for last (Christian's topics).  The committee members expend an effort just getting to the meetings to participate in the discussions.  If you're going to object to their decisions I think it only makes sense to listen to those discussions first. 

Share this post


Link to post
Share on other sites

On topic, if this is a breaking change, what difference will it make if it is put in a document labeled "4.0" vs "3.13"?  That isn't clear to me.

Share this post


Link to post
Share on other sites
7 minutes ago, Paul Wade said:

On topic, if this is a breaking change, what difference will it make if it is put in a document labeled "4.0" vs "3.13"?  That isn't clear to me.

Two main things:

  1. This is what the standard specified should happen. Making changes to the standard, which ignore explicit specifications in the standard, is... not how Standards work. It can't contradict itself. It can't specify how something is done, then do it in an incompatible way, without and still be a Standard.
  2. For the exact same reason that, I expect, the standard was set to say it should be a major version. At the moment I assume about 100% of anything using the DCS uses version 3. And it's possible to add changes to that version 3, and still work with other software using version 3, because all of the changes are additive. If there's a new label that the other device doesn't know, it can just ignore it, and ignoring it is a fine response, because it won't know what to do with it anyway. All the changes are sort of backward compatible. Updating one device/host in a lab to a new version can be ignored if there isn't a new feature needed on the other machines.
    Version 4, as a major change, means that this is all gone, as I assume was on the change from 2 to 3. If you're not absolutely sure the other machine is 4, you're not supposed to just send the same and know it will all work, sans not getting new stuff. You have to know what major version you're communicating with, for sure, unless there is also an added back-compatible way to check. It's like knowing there's one machine using the DCS and one machine using something else, which has to be configured appropriately.
    So with a change to 4, it requires manual indication of which protocol to use on what, at least until complete saturation of 4, or unless there's a good detection built-in on 4. There won't be an "The customer just updated our software on the machine, and suddenly it stops working", because the update won't switch automatically to 4.
    But with 3.13, the standard is very clear, you can't just switch to 3.13, and everything that worked before will still work. It's safe. There is no reason to add "3.13" manual configuration, or anything of the sort. By definition.

If the idea is to change how major/minor versions in the standard are handled, I suppose that's fine. I think it's a bad idea, but it's fine, and the committee can certainly decide to do that. But it didn't happen here, the definition of how major/minor versions are handled remained the same. And this is a minor version. So I can just change my line length to 255, and the Host can just fail to parse it, and everyone is working correctly, which is exactly what the definition of the major version is supposed to avoid.

 

Share this post


Link to post
Share on other sites

I understand what you're saying but I disagree.  I don't feel this is a breaking change.  A record that is invalid should be ignored.  In any event, I'll give it a day or two to see if anyone else responds to this topic in favor of postponing this until 4.0.  Otherwise the draft will proceed as described above and you'll get a chance to vote against it when the time comes.

Share this post


Link to post
Share on other sites
23 minutes ago, Paul Wade said:

You have also posted questions on the engraving reference system which is the vast majority of the discussion of both meetings.

Yes, and the last time we were actually involved in a live/phone conference about the engraving reference system, and the last time Pinney (from my company) was in a meeting mentioning it, we ended up being told it's not actually relevant for us and we shouldn't do anything with it as an engraver, and were essentially discouraged from keeping up.

So, something in the results of the discussion did turn out to appear relevant and I had some comments (to be fair mostly because it was called "Back Engraving Reference", which again opened the "are you sure it means what you say it means?" discussion, the following details are just more thoughts that came out later, since I was looking at it anyway). Was I supposed to not add anything to the discussion, even if I saw something that appeared problematic, because I wasn't at the meeting?

Isn't the purpose of these forums, and of you emailing notice about a new DCS draft, to open the discussions not only to the people who were on the  meetings?

Now, if someone from the 50 people presents on those meetings, would be kind enough to just rough timestamps of when different topics start, and post those short lists when the audio recording are out, that would be great. I'd be very appreciative, I'm sure other people will be greatly appreciative, and I'm sure it will increase the amount of people who listed to (at least part of) the meetings. As-is, again, a few minutes here and there on the forums, I can justify. Hours of listening to the meetings, most of which not directly relative to my job, would (very rightfully) get me some unkind comments from my boss.

Share this post


Link to post
Share on other sites
6 minutes ago, Paul Wade said:

I don't feel this is a breaking change.  A record that is invalid should be ignored.

So, again, what am I supposed to do with those D labels? It's valid for me now to send them longer, and it's now and invalid record for the LMS so it should ignore them. (Because, again, the idea was to ignore invalid new labels, not to invalidate active and working ones).

This is straight to the Q&A forum once the new version is out. I'm correctly sending valid data which the LMS then correctly and validly ignores, my machine no longer works after a correctly performed upgrade that fits all the requirements of the standard, what should I do?

Share this post


Link to post
Share on other sites
29 minutes ago, Yaron [LaserOp] said:

So, again, what am I supposed to do with those D labels? It's valid for me now to send them longer, and it's now and invalid record for the LMS so it should ignore them. (Because, again, the idea was to ignore invalid new labels, not to invalidate active and working ones).

This is straight to the Q&A forum once the new version is out. I'm correctly sending valid data which the LMS then correctly and validly ignores, my machine no longer works after a correctly performed upgrade that fits all the requirements of the standard, what should I do?

I didn't say "invalid record label", I said invalid "record".  So, if for any reason a system thinks a record is invalid, it should be ignored.  That includes if the field value(s) are invalid which can include too long.  With that understood, this change is no different that any other new change in a new version.  If you are 3.13 compatible but the system you're speaking to is not, you have to work that out with them.  If they are 3.13 compatible, then it won't be an issue because they should have adopted the change.  I still don't see how this is a protocol "breaking" change.  It only applies if you are 3.13 compatible.

Edit: And again, even if we call this 4.0, this problem will still exist.  I still do not agree that this is a big enough change to require a delay to 4.0 which may take years to develop as it will be a dramatic departure from the current protocol.

Share this post


Link to post
Share on other sites

By that definition, nothing is ever a breaking change.

Non-breaking changes mean that if you support the new version, but the other side doesn't, everything still works. You won't "get" the new stuff, but if it worked before the update, it will work after the update. Adding new labels. Adding fields to existing labels. Adding new optional modes/types to enumerated sets (literals/integers with a defined set of values and meanings).

When those changes happen, everything still works. If the other side is not compatible (older) then it won't be able to process/use/parse the new data. But it will still be able to process and handle whatever it knew to handle before. You can put a new Host supporting new a new version in place, and all the existing machines will still work. You can put a new machine using a newer version in place, and it will still work with the existing Host and data.

This change... makes things that previously worked, stop working, when one side is updated. It doesn't mean the older side won't get to use the new info, or won't know how to process the new data types. It makes the older side not being able to get old data from old labels that it previously knew how to handle.

D worked and was valid, now D may not work and be invalid. R worked and was valid, now it's invalid. You turn valid records, into invalid records, merely by upgrading. That's a breaking change.

If I update my machine, and it stops working ("breaks"), that's... about the definition of a breaking change.

 

How do you define "breaking change", if a change that caused something that worked before to immediately stops working, without upgrading the other side, isn't "breaking"? Does not being able to receive the exact same data, for the exact same labels, not "affect the ability of hosts and devices to communicate", as per OMAV?

 

The problem will indeed exist even when calling this 4, yes, of course. Since, again, it is a breaking change.

But switching to 4 is an indication that things can break. That's the explicit purpose and usage for a major change. If I update a machine that used 3.xx to 4, it won't (it shouldn't) start to talk 4 with other machines/hosts by default. If I update a machine that used 3.12 to 3.13, it will (and it should) talk 3.13 with other machines/hosts by default.

Things are supposed to break when switching to 4, that's what it's for. They're not supposed to break between 3.12 and 3.13.

 

Share this post


Link to post
Share on other sites

Not true.  Some of the things being discussed for 4.0 are truly "breaking" the current paradigm, like changing everything to XML or JSON.  Again, I strongly urge you to start listening to the meetings if you can't actually attend.  If you do that shortly after they are posted, and they are always posted within a few days of the meeting, you can raise your objections sooner while everything is still fresh in everyone's mind.  You can be more informed about the objections already raised, about the direction we are going for 4.0, and in general you will be able to provide more effective feedback.  Quite often objections quite similar to ones you bring up are raised and discussed, and sometimes your specific objections are raised (by me) and discussed so you could hear those responses in detail.  I mentioned you several times during the last meeting.

In any event, at this point we need to wait and see what, if anything, the other committee members say.  At this stage I still have to go with the consensus established at the last meeting.  There are a couple of other items delaying publication of the next review draft anyway so we'll have until sometime next week.

image.png

Share this post


Link to post
Share on other sites

OK, Paul, since you keep insisting, I listened (well, mostly did quick skips throughout, to locate the relevant section/s) to the recording of the VEW meeting.

The topic of the record value length was discussed roughly from 14 to 29 minutes.

  • Of these over 10 minutes, there were about 0 seconds (that I could find) dedicated to discussing whether this is a breaking change, and should it cause an increase of major version.
    There was, at the end of the session (3 hours 48 minutes), a discussion on 4.0, but it did not have any in discussion on whether the record length should be itself justify a major version, as the definition of OMAV demands.
  • So since you keep responding to my points with, essentially, "this was discussed, listen to the discussion before you raise this again" , please let me know when it was discussed, since I absolutely can't find it in the recording.
  • Also, at about 23 minutes, where most of the actual decisions were already made, mostly leaving the rest for details and bookkeeping, someone says something like "We'll put that for review, send it out, if labs have some reasonable objections" .  That... gives the impression that it should be fine to post further comments and objections, for the purpose of receiving actual feedback from the involved members, rather than, essentially, "it's basically decided, so will be voted on as-is if nobody will decide to respond".
  • New point: At around 21 minutes (though generally all over the section) there was a focus on a sub-decision to also limit the length of names of experimental/vendor-specific label names. Which, you know, technically is a valid decision, but the only reasons I could figure out was that it visually felt to some people to be too long, coupled with some snickers at the existing "Excessive label length should be avoided".
    Is that really a good reason to reduce the length of something, that may already be currently longer in practice in the field, and so possibly turning completely valid and compliant experimental/vendor-specific records under 3.12 to invalid records under 3.13?
    Sure, this might have been misused by some vendors, but the standard does say "should" and not "must", and "excessive" is open to personal interpretations. If someone created entirely valid "_IT_IS_VERY_IMPORTANT_TO_GET_THE_FULL_NAME_TO_AVOID_CONFUSION=1;2" records, just because it's "ugly" shouldn't be a reason to retroactively invalidate those. This is, again, potentially a breaking change for labs that use such labels.

Would really appreciate the timestamp where this was discussed, or response from other committee members who were in the discussion and can comment on why they want to go ahead with a breaking change (or two), instead of adding some further limits to make it non-breaking (e.g. keep old max lengths for any previously introduced records, unless device and host explicitly identified as supporting 3.13, plus way for host to identify supported version).

 

Share this post


Link to post
Share on other sites

Yaron

As an LMS vendor, I will not force any equipment to expand the fields it currently uses.  99% of all fields today follow the 80 max character limit. There are a couple of private labels that go past this limit.  I can't speak for all LMS companies but would think your engravers will continue to work the same as today. 

Share this post


Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.