PROCBLK new record + SDFMODE extension

[The Vision Council]

On 4/8/2018 at 12:43 PM, Yaron [LaserOp] said:

Sorry, accidentally posted the previous one too soon. Full one follows here:

Hi Yaron, I deleted the superfluous post.

I'll wait for Jullien to respond but perhaps just the example is confusing. I would have thought that the LMS would only send whatever ENGMARK records existed in the PROCBLK so if those other records needed to be included they would have to be in the PROCBLK as well.  To my understanding that seems to be the only way to be consistent with other record handling.  What am I missing?

[Yaron [LaserOp]]

Paul,

That would be consistent, but has a few problems and disadvantages:

1. Not so much a technical problem as a semantic one - it's not as obviously the intent as in other cases. When something tells the LMS that "GAX is X in general, and Y for device M" it's obvious that it's just Y for device M, and couldn't possibly mean anything else in any case, because any other possible interpretation is meaningless. But with records like ENGMARK it's not obvious, but rather an arbitrary decision. Because it can make sense to replace, but can also make sense to add (e.g. "ENGMARK is X in general, and Y for device M" can sensibly mean either "device M would just have ENGMARK of Y" or "device M would have ENGMARK of X and Y". it's a different case, compared to the "normal" case that is so obvious that the interpretation was more naturally assumed than discussed and decided upon.
2. A major part of the purpose of PROCBLK over PROC is efficiency. But if for multiple ENGMARK labels there is a device that just needs one more, then having to list all of the main/general ones again is counter productive.
3. PROCBLK should work essentially like a series of PROC labels, the differences being to avoid re-sending the same type/vendor/device qualifiers multiple times, and to allow data sets. But, and I think it's a very big but, if PROC interprets ENGMARK as "replace" rather than "add" then... the behavior of PROC is suddenly a bit weird, a single PROC with ENGMARK will replace *all* general ENGMARKs with the new single ENGMARK. But a second PROC, for the same device, with ENGMARK will... add to that device? That would be the correct behavior in context. Which means that already for such labels the default action is to add rather than replace.
4. Related to the previous point, again I assume the default behavior overall for these types of labels is to add rather than replace, no? What would happen on an LMS that has an existing job, with several ENGMARK records and a GAX record, if it receives a REQ=CMD with ACT=MERGE that has an ENGMARK label and a GAX label? I'd expect it will "replace" GAX, but will "add" ENGMARK. So same sort of logic should apply.
5. Unless I'm wrong in the basic assumption on the previous two? In that case, how does anything tells the LMS to add an ENGMARK (or DRILLE, or whatever other multiple-instance labels there may be) record to what it already has?
6. There are potential uses for wanting to have different ENGMARK (and such) records between device. But there are also potential uses for wanting to have a common set of ENGMARK records and then add one/some to specific devices. Which is the more common/wanted/helpful one should be a consideration.   Personally I already know of various labs who use our machines in a way to combine additional labels with what the LMS is sending the laser by default, and these pretty much invariably are used to "add" ENGMARK/ENGMASK labels, and sometimes to "replace" other labels. With PROC/PROCBLK support in the LMS this would be something that can (and probably should) be transferred to happen through the LMS, since the LMS would be capable of handling it. But automatic "replace" means that even an LMS capable of different per-device records won't be able to replace the current workarounds for sending different labels to the engraver.   Anecdotally while I'm aware of quite a few labs adding ENGMARK records this way, I'm not aware of any using the same method to replace all ENGMARK records.
7. Related, the different-records-for-different-devices ability is potentially useful enough that it will likely go beyond just initial packet from LDS when creating new job details. This is useful for integration with various management/maintenance systems and such, as a way to put additional data into the LMS. Something which I assume is already sometimes happening, just for the general for-all-devices option (e.g. using REQ=CMD with ACT=MERGE or upload packets, or whatever). Expanding those to include PROC/PROCBLK seems like a natural usage for the new feature. And again, for multi-instance labels it can make a lot more sense to add for specific cases, rather than replace.
8. Notice that for the original main LDS packet, both possibilities are doable. to "add" when the default is "replace" they just, as you mentioned, re-send everything to each device. And to "replace" if the default is "add" they just avoid sending these labels in the general section and duplicate for all relevant PROCBLK types. In either case the "do something differently from the default" would be more verbose, so it's probably better to make the default the more common case.   But, for any other device/software modifying job data later, it's not possible to "add" if the default is "replace", and not possible to "replace" if the default is "add". So the very important question is what are such devices/programs more likely to want to do? Which again, I think (and know at least for ENGMARK based on experience) would be "add".

[Etienne JULLIEN]

Greetings Yaron,

Once again, thank you for your acute awareness and relevant concerns.

My first reaction to your initial post was the same as Paul's, as it seems to be the only way to address the problems you mention with the current definition.

As you stated in point #3, this is probably a discussion we should have had as soon as the PROC record was introduced, since it requires distinct behavior depeding on the record type (singleton / collection) and the LDS instructions (replace block / add / replace specific entry).

One idea could be to use a different separator (such as '+') when we want the LMS to "add".

But I feel this would require a lot of thinking to make sure it covers all cases.

Since we have a non-optimal workaround with the current definition (LDS needs to re-specify all collection entries in each block), I would suggest that we let this one out in 3.12, and try to draft a more optimal solution here, to be validated during the next VE, if that's ok with you.

 

[Etienne JULLIEN]

Sorry to reply to myself, but the editing policies are quite strict around... 

2 hours ago, Etienne JULLIEN said:

Since we have a non-optimal workaround with the current definition (LDS needs to re-specify all collection entries in each block), I would suggest that we let this one out in 3.12, and try to draft a more optimal solution here, to be validated during the next VE, if that's ok with you.

This does not sound as clear as intended upon reading with fresh eyes, so please replace it with

Quote

Since we have a non-optimal workaround with the current definition (LDS needs to re-specify all collection entries in each block when specific values are needed for this device), I would suggest that we let the current version be included in 3.12, and try to draft a more optimal solution here, to be validated during the next VE, if that's ok with you.

 

[Robert Shanbaum]

It would appear that some elaboration of the definition of PROC records is called for.  I think that my own interpretation has been that for records that can appear multiple times in a packet (such as ENGMARK or DRILLE) the operation of a PROC record would be to add the device-specific records to the records sent to the machine.  We don't have a way to suppress sending particular records of that kind to a machine, but it's not clear to me that such a feature is necessary (and if it is, it will be challenging to implement in a robust way, requiring matching strings of characters across multiple "device versions" of a record).

The more general case - wherein records are "singletons" (that is, only one record having a given label appears in any packet sent to a machine) - the usage is clear, is it not?

I don't understand the comment above regarding a "non-optimal workaround"...  

[The Vision Council]

1 hour ago, Robert Shanbaum said:

It would appear that some elaboration of the definition of PROC records is called for. ...  

I was thinking along the same lines.  If using just PROC like this:

ENGMARK=MASK;MainDesign;....
ENGMARK=TXT;A;....
ENGMARK=TXT;B;....
ENGMARK=DCS;ADD;....
...
PROC=ENGMARK;ENG;;;TXT;D....
PROC=ENGMARK;ENG;;;TXT;E....

Is that additive so that 6 ENGMARK records are sent to the engraver?  Or is it replacing ENGMARK so only the two PROC ENGMARK records are sent?  What was the intent?  If we define that then the definition of PROCBLK needs to match, doesn't it?

[Robert Shanbaum]

We didn't expressly address in the standard device-specific records that can appear multiple times; but as I wrote, I always figured these would be additive.  It may be that we should expressly identify such records ("repeating records") - I can tell you that we have to do that in our software.

[Etienne JULLIEN]

16 hours ago, Robert Shanbaum said:

I don't understand the comment above regarding a "non-optimal workaround"...  

Good morning Robert,

The option we were considering before you made clear that PROC records were to be additive in case of "repeating records" was that they would totally replace default values.

Hence, to send an additional ENGMARK to a specific device, you would need to declare again all the default ones:

DO=B
ADD=1.75;2.50
ENGMARK=MASK;MainDesign;....
ENGMARK=TXT;A;....
ENGMARK=TXT;B;....
ENGMARK=DCS;ADD;....

PROCBLK=1;ENG;VEN1;...
1!ENGMARK=MASK;MainDesign;....
1!ENGMARK=TXT;A;....
1!ENGMARK=TXT;B;....
1!ENGMARK=DCS;ADD;....
1!ENGMARK=MASK;ExtraVen1...

But since it forces to duplicate all the default engravings for each device that requires an additional/different one, it did not sound like an optimal solution.

 

Now, I can see 3 choices for repeating records:

• We stick with the "Replace" global behavior (as soon as you have a hit for a record on a given context - DEV+VEN+MODEL+S/N, or DEV+VEN+MODEL, etc. - , you take all entries of this record on that context, and ignore the other context levels)
• We can let the LDS give specific instructions (delete, replace, add)
• We can have a "by definition" behavior such as the one you are suggesting

First option is pretty simple, both for LMS and LDS, drawbacks being records duplication.

Second option would require us to define a much more complex syntax than the current PROC.

Third option would require to explicitely tag all "repeating records" as such in the DCS, and have the LMS aware of that particular characteristic (so that they can process the PROC record accordingly).

 

Option one strikes me as the easiest way to go, but it sure is more brutal and less graceful than the third one.

 

 

 

 

 

 

 

 

 

 

 

 

[Yaron [LaserOp]]

The problem with option one (everything is "replace", including non-singleton labels) isn't just record duplication and so it being less graceful. That is indeed the problem, but only for the case of an initial upload of data for an empty (at least of related label) job.

But for all other scenarios (e.g. LDS uploading data to a job that already has some initial records because it was created earlier by the LMS or another customer/lab system, or alternately any other device that is modifying job data - which is something that can and does happen and is supported by the standard) this is not the case. Because in all such cases the machine that sends/uploads the data knows what it wants to send, but not what is already in the LMS for that job.

Which brings back the same issue. If something wants to modify a singleton (BTW, Robert, thanks, I think it's a good term for this) label, pretty much by definition it wants to replace it, and this only happens when it "knows" that what is was before doesn't matter and should be replaced. But for a repeating (not that crazy about this term, but usable) record this isn't the case. It doesn't have to be the case, and I of course think similarly to Robert that it's not even the usual/normal/common/expected case. For these repeating records it's much more likely (though my opinion is again admittedly skewed by familiarity mostly with ENGMARK) the desire is to add something rather than overwrite/replace all other instances (which may have come from several different sources anyway).

-

It's true that the third option (default behavior is "replace" for singleton labels but "add" for repeated ones) will require LMS (and other devices sending/using them) to know which label is singleton and which isn't.

But... they sort of have to do that now anyway, even if it's not given an explicit term in the DCS. Because an LMS has to know how to handle these anyway, and knowing how many copies are valid is already a part of that.

So it shouldn't put an extra effort on anything/anyone, it's an effort they already have to do. Again, if an LMS receives a job update with e.g. GAX it already must know it's a singletion, and if it receives one with e.g. ENGMARK it already must know it isn't.

It's not just an issue with PROC, it's an issue with regular job data updates. PROC just makes it more important that the standard will be explicit in the handling, rather than having people make implicit assumptions as must have happened so far (e.g. just like Robert's LMS). The logic of how it happens on PROC must match the one of how it happens without PROC. Which, again, is very probably adding repeating records and not replacing them (which is for sure what is happening in real-world cases with ENGMARK. I personally don't know about real-world cases with other repeating records, but would assume similar simply by the fact that they are indeed repeating).

[The Vision Council]

In a contrived example, we have three engravers.  Two high resolution laser and one lower resolution mechanical.  We have a complicated lens feature which can only be engraved on the high resolution engravers.  Which of the following is the way we'd want this handled if we used only PROC?

Method 1 - Non-cumulative
=========================
ENGMARK=Complicated feature
ENGMARK=Plain feature 1
ENGMARK=Plain feature 2
PROC=ENGMARK;Dumb Engraver;Plain feature 1
PROC=ENGMARK;Dumb Engraver;Plain feature 2

Method 2 - Cumulative
=====================
ENGMARK=Plain feature 1
ENGMARK=Plain feature 2
PROC=ENGMARK;Smart Engraver 1;Complicated feature
PROC=ENGMARK;Smart Engraver 2;Complicated feature

In both examples we want both plain ENGMARK records to get to the dumb engraver but to have the complicated ENGMARK only go to the smart engravers.  The first method is non-additive so that we have to declare the features explicitly.  The second is cumulative where we only include the additional features.  In the first it would be easy to exclude non-singleton values but in the second it's more difficult and granular I think.  I would prefer to work with the first method I think.  Despite it's higher verbosity I like the explicit nature of seeing exactly what will go to "Dumb Engraver" without having to check the other records.  It's how a singleton record behaves as well so it's consistent. 

This is a tough one.

[Yaron [LaserOp]]

Paul,

First, regarding:

Quote

I like the explicit nature of seeing exactly what will go to "Dumb Engraver" without having to check the other records.  It's how a singleton record behaves as well so it's consistent.

That's not entirely correct. Or, well, it depends on how you look at it. A singleton record will behave that way, on account of being a singleton. When you send PROC=SBFCIN you replace ALL of the SBFCIN record. But you do that because there is only one SBFCIN.

But with a non-singleton records, they may all have the same label name, but they're different entities. So bunching them together is not unlike saying that, for example, SBFCIN and SBFCUP are very closely related, so having a PROC with just SBFCUP should also remove (zero) an upper/general SBFCIN. That idea sounds immediately wrong, because they have the same name, but they do come together and serve a closely-related purpose. Non-singelton records are sort of like that too, they do share a name but they're otherwise different entities representing different things.

ENGMARK="Plain feature 1" is not the same as ENGMARK="Plain feature 2", these are two different individual records, doing two different things.

Or, better example, since it's sort-of-not-entirely-singleton, what about most datasets? consider:

TRCFMT=1;128;E;R;F
R=...
TRCFMT=1;128;E;L;F
R=...
PROCBLCK=1;ENG;....
1!TRCFMT=1;240;E;R;F
1!R=...

If an engraver later asks the LMS for frame trace, for the right side there would be one based on (of course currently possibly interpolated otherwise) the 240 points trace.

For the left lens sent to an engraver... do you think there should be one based on (and again possibly interpolated) the 128 points trace? or do you think there should be no trace sent?

What seems to me like the natural expected answer is that the engraver would receive frame traces for both eyes, one from the 128 source and one from the 240 source. But, well, with the "replace for non-singleton" attitude, TRCFMT has been replaced for engravers. So engravers only have TRCFMT explicitly set for them. Which has a right trace, but doesn't have a left trace.    This feels like the wrong interpretation. But it follows the same logic for labels like ENGMARK, the only difference is that there can be N copies of ENGMARK but  2 copies of TRCFMT. But, again, both these two TRCFMT copies are called TRCFMT, so...

I guess you probably agree that both TRCFMT datasets are distinct, even though they both use the same names of labels. And agree that replacing that dataset for one eye in PROC wouldn't affect the identically-named labels for the other eye. So why is that different for ENGMARK, or for DRILLE datasets, or other non-singleton labels?

[Yaron [LaserOp]]

Second, this still focuses on when everything comes from a single source. That is, in your example there are no ENGMARK records on the LMS before this data packet.

But what when there are? Suppose the LMS is set to always engrave some feature the lab wants, so when a job is created it automatically populates it with "ENGMARK=Lab feature 1". (Let's even assume it's for area that will be removed by the edger, so no issues with other agreements/expectations by some LDS vendors on what will be engraved)

Then one of the LDS vendors in the lab have their own engraving layout, so when the LMS asks them to calculate the job, they send a packet including ENGMARKs as in your example (either case).

What should one of the "basic engravers" receive from the LMS?

Option 1:
----------
ENGMARK=Plain feature 1
ENGMARK=Plain feature 2

Option 2:
----------
ENGMARK=Plain feature 1
ENGMARK=Plain feature 2
ENGMARK=Lab feature 1

I'd expect option 2. I get that Robert would also expect option 2.

That's the basic difference of receiving data about pre-existing labels, between the singleton and non-singleton ones.

So if even without PROC singleton labels should be replaced but non-singleton labels should be added, then providing more of then through PROC should follow the same process, because that is what is already being done without PROC.

-

As an aside, of course a problem with this "add" attitude for non-singleton labels is that there is no obvious way to replace/delete such labels if needed. But similarly with the "replace" attitude for non-singleton labels there is no way to add them. And not being able to add such labels (something that is useful and in demand and is already being done with workarounds) is a much much bigger problem than not being able to replace them (which seems less often useful, and which maybe can be easily provided by something else, like maybe an additional ACT to REQ=CMD for clearing specific labels? Can probably wait until and if someone actually asks for it. Or maybe allow to force-clear a non-singleton label by sending it empty to the LMS, the downside being that this will have to be ordered but at least for PROCBLK everything is already decided to be ordered ).

[Robert Shanbaum]

Yaron, we only send one tracing per packet, ever.  And tracings are a special case, because I'm pretty sure that all LMSs can send whatever number-of-points or formats any device might specify in its initialization, without requiring PROC records (or PROCBLK datasets) at all.

As I wrote, the standard is not presently explicit when it comes to which records (or datasets) might be repeating, but it seems to me that there's not really much doubt about which ones are, and which aren't.  Can anyone name a record for which that is not true?

I do not agree with the proposition that the presence of one record in a PROC record implies that another, different record should be changed.

[Robert Shanbaum]

I suppose that it's possible that a use case could exist wherein a set of repeating records might need to be replaced by another set of repeating labels (or, possibly, just one label - an ENGMASK might replace a set of ENGMARKs, for instance).  I'm not sure how likely such a case is to obtain.

[Yaron [LaserOp]]

Robert,

Sorry if I wasn't clear.

I used tracing as an example to illustrate that there already exist cases of labels which can have different distinct instances even though technically these different instances have the same name. Of course traces aren't ever expected to be handled that way. But they can show the concept of having a label with the same name, passing the same sort of data, while still being distinct from another label of the same name, where pretty much everyone would agree wanting to modify one does not affect the other. (e.g. if you won't erase all existing traces if you receive just a new one, why would you erase all engravings or drills, when you receive a new one?).

And we are already in agreement that having one record present shouldn't affect another different record. That, again, was the point I was trying to illustrate, that in practice non-singleton records like ENGMARK are all also different, even though they have the same name, so having one like that in a PROC shouldn't mean the others should be changed.

[The Vision Council]

I'm going to be setting up a conference call.  I emailed a few of you to get a consensus on the time and then I'll send a poll to the committee.

[Etienne JULLIEN]

After carefully reading all the posts, I'm inclined to agree with Yaron.

LMS obviously already have "intuitive" knowledge of the "repeatable" characteristic of a record.
And they already have rules to handle records depending on this characteristic (Replace "GAX" or Add "ENGMARK" for example).

So it makes sense to keep it simple and use the same ruleset for PROC records.

Maybe the only thing we need, really, is for this repeatable characteristic to be explicit in the DCS records definitions.

[The Vision Council]

So then perhaps a call is not necessary.  Please let me know if we have consensus on the following:

1. Modify the definition of PROC to explicitly describe the behavior of non-singleton records like ENGMARK.  Namely, that it is additive or cumulative in nature and that if a distinct set of non-singleton records is desired to be sent to a specific device then they must be entirely contained in PROC records.  Probably should include some examples.

2.  Ensure that PROCBLK's definition matches the defintion of PROC with regards to non-singleton records.

Agreed?  Or am I still missing the plot?

[Yaron [LaserOp]]

I think this is the current consensus on the discussion here. Though if any silent readers of this thread have some contrary experience with handling of such labels, now would be a good time to mention it.

 

I also agree with Etienne that it's probably then a good idea to make the singleton/repeated characteristic be made into explicit defined terms in the Standard. This is a Standard, so if this characteristic is explicitly referred to by something (which it will be now), then it needs to be explicitly and clearly defined. I do not think, however, that there is a need to then go and explicitly list in the DCS for each and every record how it falls (or make a list), as that would be very easy to infer for each record based on the definition.

Related, in addition to mentioning this in PROC/PROCBLK, the same note should probably be in other methods of sending data to the LMS. Since the expected (but currently implicit and unlisted) behavior there is the source of assuming this should also be the behavior in the context of PROC, and LMS are probably acting this way anyway, this is a good opportunity to make it explicit in the standard as well.