Paul Wade

LPDS 0.78 DRAFT

Recommended Posts

Dear Colleagues,

Please find attached to this post the draft of the new Lens Product Description Standard.  Please post all comments, questions, suggestions and critiques in this thread so that we can keep the process transparent.  If you have specific concerns you are not comfortable addressing publicly please feel free to email them directly to me.

Regards,

Paul

TVC Lens Product Description Standard 0.78-DRAFT.pdf

Share this post


Link to post
Share on other sites

Fixes for the example in section 4:

  • Use the standard double-quote char everywhere - some non-standard ones are in the example (e.g. “Product IDs” versus "Product IDs")
  • "Blank Collections" section - remove the last comma after BLK-YYY-010 since this is the last element in the array

I attached a working file after the fixes above.

Suggestions for consistency:

  • Consider renaming UsableDiameter to have a space - same for RightOPC, LeftOPC, RightID, etc. as the convention seems to be space separated names like:
  • Quote

    "Product Name": "Example Product",
    "Product ID": "PRD-YYY-000001",
    "Design ID": "DSN-YYY-000001",

  • Capitalize words in the names - change "Owner name" to "Owner Name" and "contact" to "Contact"

Clarity on JSON Arrays

In section 7.3

Quote

Please note that due to limitations within various JSON parsers, the order of the elements within the Lines array are not guaranteed...

RFC 7159 says "An array is an ordered sequence of zero or more values."  So, something like [1, 2, 3, 4, 5] must always be in that order.  If a JSON parser does not maintain that order, then the parser is faulty.
 

lpds-example.json

Share this post


Link to post
Share on other sites

Section 7.3 for Sag Charts
Rather than have Lines be an array of strings, why not do like:

"Lines": [  { "Base": 6, "Add": 1, "Radius": 15, "Angle": 0, "Sag": 3.17 }, {  "Base": 6.00, "Add": 1.00, "Radius": 15.00, "Angle": 15, "Sag": 3.56 }, ... ]

No need to multiply by 100 and naming the values gives more clarity.

Share this post


Link to post
Share on other sites

To the entire LPDS Committee, thank you for all your efforts. This is looking really good. Consensus standards are very important for our industry and this is an excellent representative of the great work that you all do!

3.1 - JSON format - Us JSON the proper format to use for this standard and do we believe all manufacturers will have the technical expertise to write this in JSON? They have historically used a csv format for the LDS and this may be very cumbersome for them.

5.1 - Remove the highlight on Violet

5.2 - Abbreviation List - Did we pull the most resent abbreviation list for this?

5.4 - Materials - How are the LMS systems addressing Trivex? (OP?) We should not have a brand name in there but we may want to find an abbreviation for a 1.530 index lens?

5.5 - Design Styles - We still have progressive lens as a design style. I suggest we also have next to it (power variation) as this is the term that ISO uses and ANSI will have in its upcoming revision. This would also be the same term used for "degressive, office, power boost, etc". Maybe the lens manufacturers could chime in here?

6.1 - Describing Compressed Base Curve Charts - the current wording in the second paragraph is; "The following bottom fragment of a base curve chart is for a known add power, power, with the vertical axis representing the prescription sphere power and the horizontal axis representing the prescription cylinder power." Is the area I have changed the text color correct?

Share this post


Link to post
Share on other sites
On 10/16/2019 at 5:34 AM, Michael Vitale said:

3.1 - JSON format - Us JSON the proper format to use for this standard and do we believe all manufacturers will have the technical expertise to write this in JSON? They have historically used a csv format for the LDS and this may be very cumbersome for them.

 

I have always been more in favor of using XML for this standard.  There are three reasons, in order of increasing importance:

1.  XML does not have the idiosyncracy of inconsistent array ordering, which affects JSON.  This is currently constraining our design options for base curve chart data and could have the same effect on future development.

2.  XML is more human readable than JSON, especially for large, complex documents such as these catalogs.  Although we will end up parsing these files with software, people are still going to need to read them for troubleshoooting and support purposes.  XML is better suited for that.

3.  Using XML will allow us to encode all the rules of the standard in a single XML Schema Document (XSD).  This XSD can then be used in most development platforms to automatically generate libraries of code for parsing, writing and validating XML files.  There are numerous online document validators that if fed an XSD and a sample instance XML file, they can validate the instance document is properly formatted and follows all the necessary rules.  This will enable much faster development and adoption of the standard.

I have not heard any good arguments for using JSON.  It mostly seems to be "XML is old and stodgy, JSON is new and fun".  I think XML is a much better language for this type of endeavor.  We can also always convert from XML to JSON for transmission purposes, but defining things in XML gives us many advantages.

My understanding was that the decision on the language used would not be made until the standard had been fleshed out.  I had been pushing my points about XML back when Daniel was still in charge and I recall that we specifically tabled that conversation until we had defined all of the fields.  I was not expecting it to be dictated in the standard yet.

Share this post


Link to post
Share on other sites

Thanks, Dave.  

On 10/15/2019 at 5:06 PM, Dave Wedwick said:

Fixes for the example in section 4:

  • Use the standard double-quote char everywhere - some non-standard ones are in the example (e.g. “Product IDs” versus "Product IDs")
  • "Blank Collections" section - remove the last comma after BLK-YYY-010 since this is the last element in the array

I attached a working file after the fixes above.

Suggestions for consistency:

  • Consider renaming UsableDiameter to have a space - same for RightOPC, LeftOPC, RightID, etc. as the convention seems to be space separated names like:
  •  
  • Capitalize words in the names - change "Owner name" to "Owner Name" and "contact" to "Contact"

Clarity on JSON Arrays

In section 7.3

RFC 7159 says "An array is an ordered sequence of zero or more values."  So, something like [1, 2, 3, 4, 5] must always be in that order.  If a JSON parser does not maintain that order, then the parser is faulty.
 

lpds-example.json 4.85 kB · 0 downloads

Yeah, Word was occasionally throwing in that alternate quotation, I'll have to see what that's about, as it'll cause problems on future updates.

I agree with the capitalization changes.  

For the parser concern, I know I'd personally run in to an issue with a parser not maintaining the order of the array, and thought I'd determined at that time that it was a caveat of using JSON.  But, yes, if it's part of the standard, that should be sufficient to warrant having parsers that don't fiddle with the order of the array.  Thank you!

Share this post


Link to post
Share on other sites
On 10/15/2019 at 5:06 PM, Dave Wedwick said:

Section 7.3 for Sag Charts
Rather than have Lines be an array of strings, why not do like:

"Lines": [  { "Base": 6, "Add": 1, "Radius": 15, "Angle": 0, "Sag": 3.17 }, {  "Base": 6.00, "Add": 1.00, "Radius": 15.00, "Angle": 15, "Sag": 3.56 }, ... ]

No need to multiply by 100 and naming the values gives more clarity.

The use of the lines done as a single string was simply to compress the amount of space the base curve charts will take up, given the number of records.  I'm not entirely opposed to bringing them out to objects instead, but I'd wager that someone looking at interpreting the sag chart will likely be doing so after an import into a system where the information could be presented in a friendlier manner.  

Share this post


Link to post
Share on other sites
On 10/16/2019 at 8:34 AM, Michael Vitale said:

To the entire LPDS Committee, thank you for all your efforts. This is looking really good. Consensus standards are very important for our industry and this is an excellent representative of the great work that you all do!

3.1 - JSON format - Us JSON the proper format to use for this standard and do we believe all manufacturers will have the technical expertise to write this in JSON? They have historically used a csv format for the LDS and this may be very cumbersome for them.

5.1 - Remove the highlight on Violet

5.2 - Abbreviation List - Did we pull the most resent abbreviation list for this?

5.4 - Materials - How are the LMS systems addressing Trivex? (OP?) We should not have a brand name in there but we may want to find an abbreviation for a 1.530 index lens?

5.5 - Design Styles - We still have progressive lens as a design style. I suggest we also have next to it (power variation) as this is the term that ISO uses and ANSI will have in its upcoming revision. This would also be the same term used for "degressive, office, power boost, etc". Maybe the lens manufacturers could chime in here?

6.1 - Describing Compressed Base Curve Charts - the current wording in the second paragraph is; "The following bottom fragment of a base curve chart is for a known add power, power, with the vertical axis representing the prescription sphere power and the horizontal axis representing the prescription cylinder power." Is the area I have changed the text color correct?

Hi Mike, 

3.1 - It'll likely warrant a vote of the group between JSON or XML, given Paul's concerns.  I wouldn't want to venture back to CSV, as that's really limiting in terms of extending the standard out in the future to cover currently-unforeseen issues, nor is CSV great for describing this inheritance and relational catalog.  But it's definitely not going to be something that could be produced in Excel, which is going to cause some headaches for folks. 
5.1 - Done
5.2 - It was pulled from the Lens Description Standard 2.2 - if there's a newer list, please send that along and I'll incorporate it there.
6.1 - Apparently I was worried about it not having enough power.  I've removed the excess now.   

I don't have anything to say about 5.4 and 5.5 yet.  Thank you!
 

Share this post


Link to post
Share on other sites
3 hours ago, Paul Wade said:

I have always been more in favor of using XML for this standard.  There are three reasons, in order of increasing importance:

1.  XML does not have the idiosyncracy of inconsistent array ordering, which affects JSON.  This is currently constraining our design options for base curve chart data and could have the same effect on future development.

2.  XML is more human readable than JSON, especially for large, complex documents such as these catalogs.  Although we will end up parsing these files with software, people are still going to need to read them for troubleshoooting and support purposes.  XML is better suited for that.

3.  Using XML will allow us to encode all the rules of the standard in a single XML Schema Document (XSD).  This XSD can then be used in most development platforms to automatically generate libraries of code for parsing, writing and validating XML files.  There are numerous online document validators that if fed an XSD and a sample instance XML file, they can validate the instance document is properly formatted and follows all the necessary rules.  This will enable much faster development and adoption of the standard.

I have not heard any good arguments for using JSON.  It mostly seems to be "XML is old and stodgy, JSON is new and fun".  I think XML is a much better language for this type of endeavor.  We can also always convert from XML to JSON for transmission purposes, but defining things in XML gives us many advantages.

My understanding was that the decision on the language used would not be made until the standard had been fleshed out.  I had been pushing my points about XML back when Daniel was still in charge and I recall that we specifically tabled that conversation until we had defined all of the fields.  I was not expecting it to be dictated in the standard yet.

1) Apparently I've been under an incorrect misconception about the ordering issue, as pointed out by Dave.  It might be time for me to look for a new parser that doesn't fiddle with things.
2) I'm not really sold on the idea that XML's any more human readable than JSON - in fact all those extra closing tags visually clutters things up for me. 
3) There's JSON Schema that corresponds to XML's XSDs, http://json-schema.org/ and an online validator at https://www.jsonschemavalidator.net/

 

A big reason to use JSON is its brevity, and native ability to handle arrays, of which we're going to have a fair number in the Standard.  

 

Nabbed the following example from https://sentai.eu/info/json-vs-xml/

JSON Example 1 
{“employees”:[  
    {“name”:“A”, “email”:“a@a.com”},  
    {“name”:“B”, “email”:“b@b.com”},  
    {“name”:“C”, “email”:“c@c.com”}  
]}  

The XML representation of above JSON example is given below.
<employees>  
    <employee>  
        <name>A</name>   
        <email>a@a.com</email>  
    </employee>  
    <employee>  
        <name>B</name>   
        <email>b@b.com</email>  
    </employee>  
    <employee>  
        <name>C</name>   
        <email>c@c.com</email>  
    </employee>  
</employees>  

Share this post


Link to post
Share on other sites

That is a very simplistic example.  I agree that for short data sets there is value in brevity and this is exactly the type of data where I would go with JSON.  The catalogs will not be small data sets.  Reading a large mass of data and trying to figure out where the closing tag is in JSON is much more difficult than XML.  Additionally, almost all text editors that aren't Notepad can automatically prettify XML and will do syntax highlighting making viewing the data much easier than your black on white example.  There are also many more tools already available for dealing with large XML datasets by hand than I've been able to find for JSON.  I'm still not convinced, especially by such a contrived example. that JSON is better for this project.

Also, the JSON schema standard is still in draft stage.  The XML schema standard is well established.

I would further argue that as a committee we have more expertise available in XML than in JSON.

And what about code generation from XML schema's?  That is a huge benefit IMO. 

We seem to by trying to force JSON to fit, again, because it's newer and sexier, not because it offers any true benefit.

JSON is awesome, just not for this.

You might find this interesting.  It's a fair assessment despite the biased sounding title. https://codepunk.io/xml-vs-json-why-json-sucks/

Share this post


Link to post
Share on other sites

I'm not too sure about the use case you specify, where it's more readable due to being able to find a closing tag.  Is someone going to be scrolling through a list of <Products> until they get to </Products> and have determined much, as opposed to searching for what they're actually interested in?   I would expect the main text editors in use in the major OS's out there can format JSON as well as XML.  And I'm still not sure we want anyone having to deal with changing much of the payload of this thing directly, in any event.   

It's likely my own unfamiliarity with XML, but I find his examples at the link you posted seem designed to demonstrate problems with JSON that aren't just problems with JSON.  In the example where it's showing a question going from a single correct answer, to being multiple choice, it seems to suggest that for something processing XML to accept the change the only thing you'd have to do is add another element, as opposed to JSON which would switch to an array.  Except the processor is going to have to switch in either case to handling the possible answers as an array.  And given the 'strict specification' of only having one correct answer before, this really seems designed to make it easy on the XML side by presupposing you'd have a 'correct' attribute on the element already, so that you could more easily slide in a second one with a different value for that attribute because it's presumably already looking at that.  Not including my favorite part, which is his closing tags don't match the opening tags, which can obviously happen in XML (not that you couldn't forget a comma or over-include one in JSON).   

I haven't used a code generator for XML, or XPath, though I can certainly see how both would be handy.  

I certainly don't want this to be something that slows down the standard from getting used, so if you feel XML's going to cause parties to adopt it more quickly, I'm fine with that.  I think a lot of the functionality that's present in one, is present for the other, even if in draft form (where apparently JSON schema has been percolating for 10 years judging from https://datatracker.ietf.org/doc/draft-handrews-json-schema/).  And I think anyone that's capable of working with one is capable of working with the other. 

I don't mind going through the document and switching it to be in XML.  Can it just go up for a vote of the group?

Share this post


Link to post
Share on other sites

I concur that we will eventually need to put this to a vote.  How would you like it to be presented?  Simply " Do you prefer XML or JSON?" or with talking points?

As for my position, and what I took from the various articles and white papers I've read on the subject, it's really simple.  If you're exchanging data between disparate systems, especially non-persisted data over a network interface (such as web services) where terseness is important, and you're using name/value pairs or simple data structures, then JSON is the clear and obvious choice.  On the other hand, if you're exchanging data between database systems using complex data structures in large, persisted data sets, then XML is the clear and obvious choice.  While in both cases one could use the alternative language, doing so is like pounding in a nail with the ball side of a ball peen hammer.  You may eventually get the job done but striking the nail cleanly and consistently is more a matter of luck than intention.  I'm merely advocating for using the best tool for the job, and to me that is clearly XML.  When we start talking about DCS, however, then I'd be more in favor of JSON because terseness is a factor there and we are already dealing with simple data structures and name/value pairs.

In any event, let me know how you'd like to present the options and I will create a poll.

Share this post


Link to post
Share on other sites

I'd wager that including a reference to this thread would be a sufficient basis for talking points, so the poll could simply be for picking XML or JSON.  Or do you mean allow a forum conversation about it?  Because we'll certainly want folks to be able to speak out. 

I don't know if there'll be an argument of, "Why do we have to pick one at this point," but to that I'd say that there's enough uniqueness to how the information would end up presented between the two formats that the Standard really needs to be specific.  Trying to deal with both would just be messy and confusing,.   

 

22 hours ago, Paul Wade said:

If you're exchanging data between disparate systems, especially non-persisted data over a network interface (such as web services) where terseness is important, and you're using name/value pairs or simple data structures, then JSON is the clear and obvious choice.  On the other hand, if you're exchanging data between database systems using complex data structures in large, persisted data sets, then XML is the clear and obvious choice. 

That's just it - Ideally, I'd like the transmission of the payload to be a web-based API for retrieving some new set of products.  So that software vendors could point to a manufacturer's or designer's API, retrieve new products in the Standard's format, and be done.  Or new items could be presented on a web page, based on a catalog delivered via the Standard.  Things are only getting more webby, and I think we'd be missing the mark if we're not targeting making these catalogs available through web services, in addition to passing around files.  

As for the complexity, though we'll have a lot of repeated elements (multiple products, parameters, etc) and I'd expect an individual payload could be quite large, the actual elements involved are not complex compared to the allowances of XML.  I expect any of the complexity in the structure about which someone will have difficulty understanding will most likely be due to the amount of information contained in the Standard, and how the pieces relate to each other.  There's not some inherent property of XML that's going to explain how a Blank Item relates to a Product.  If we do go with XML, we'll need to determine which properties are elements, and which are attributes of elements, and then explain why they're like that, and make sure that every time we need to add some new property we have that same conversation.  The fact that JSON has a more limited set of things it can do simplifies this.  

You mention that JSON would make more sense for a successor to the DCS, but terseness seems like the only relevant factor there.  I would imagine part of a redesign of the DCS would be to expand on its capabilities, and that would likely involve making use of some number of objects in the file.  At what number of objects, amount of nesting, or length of file, do you determine that it's simply too complex for JSON and should instead be XML?  We save these files out for analysis and can attach them in emails, so the fact that the file can stick around on a hard drive shouldn't be it.  

But at this point we're rehashing old ground - you're telling me it's obviously the best choice, I'm not seeing the obviousness, and we could likely go around and around until there's some new third option that we both think is trash.  I think we just need a poll, get more people in the conversation, and move on.

Share this post


Link to post
Share on other sites

A web based API would be nice, and there is nothing to say that the transmission method for that API couldn't be converting the XML data to JSON (which works well when XML is the starting data and you have a nice schema).  However, the reality is that such a service would be at least a few years away from a published standard and even if we have that tool, many of us who directly support the folks creating these files are going to be dealing with physical files and not web services.  You seem to be under the impression that all lens companies use databases to track this stuff.  As I've tried to explain during the meetings, that is simply not reality.  Have some conversations with Tania or Dave Wedwick, et al, about the type of people they refer to me for help with their lens data.  I promise you we will be handling, reading, and troubleshooting lots and lots of text files.

Also not every company is going to want to submit their files to centralized DB.  We have already proposed this in the past for LDS data, which is much less proprietary, and some larger manufacturers were not interested.  So, in the end, we're going to be sharing files via email and such more than pulling things automatically from some web service which hasn't even been defined yet.

As for complexity, with a well defined XML schema such complexity is far more easily explained and documented (not to mention the schema will prevent mistakes).  The schema itself can be self-documenting.  And again, the XML Schema standard is well established while the JSON schema standard has been languishing in draft stage for many years.  There are also several tools that can take an XML schema and generate a nice document showing all the links between the elements and attributes that can be directly used in the standard.  Essilor has done this with RxML.  I'm not aware of any comparable tools for JSON.

As for DCS, terseness is the only factor that gives JSON an edge in DCS, as I wrote.  I wouldn't object to XML either for the flexibility and, again, the benefits of using XSD.  However, if there was a proposal to use JSON for DCS I would not object because it makes sense in that context (real time data exchanges, such as happens in web services) and in reality 95% of the records are simple enough that complex types and other XML types aren't really necessary.  JSON was never intended for the type of data we're defining here.  Hence the reason the schema definition is taking so long IMO.  They are trying to make it as functional as XML when it was never intended to be a replacement for XML.  Rather, a supplement to XML specifically for the purpose of web services.  After all, the "JS" stands for JavaScript.  Again, you can force that square peg into the hole if you want, but it seems like a poor way forward compared to using the correct tool.

Most of your counterpoints seem to be "Yes, XML does that but so does JSON".  I would argue that in every case other than terseness XML does it better and there are still no factors that make JSON a better choice, only a preferential one, e.g. being more familiar with JSON which to me should not be a factor when writing standards we expect to be adopted internationally.  I've worked with both JSON and XML extensively and I would never pick JSON for a project like this because it is simply not the best tool for the job and this is backed up by plenty of information in white papers and tech articles.

In any event, as you said, it is apparent we are not going to reach a consensus betwixt ourselves so I'll post the poll with a link to this thread.  The poll automatically allows people to post comments so they can state their reasons.

Share this post


Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.