Warning: your electronic patent search databases have gaps!

[tweetmeme source=”Intellogist” only_single=false]

UPDATE: For a further enlightening discussion of the gaps in the USPTO full text database, please see the comment section of this post (click the word “Comments” where it appears at the very end of this post).

Recently, a message came over Carl Oppedahl’s PAIR discussion list highlighting a mysterious gap in the USPTO’s online patent database: data seemed to be missing for patent numbers between 6,363,527 and 6,412,112.

Rick Neifeld, of Neifeld IP Law, responded that his 1999 survey into the PTO’s data revealed many errors in the USPTO’s data, as many of us have probably suspected for some time.  Rick’s description of these errors is very interesting:

The dirt consisted of things as minor as numerous misspellings of assignee names, or HTML pages non compliant with HTML standards, to HTML text that could not be deconstructed into component sections due to HTML formatting errors, assignment records that were combined, corrupt, unreadable.

I absolutely expect our current crop of electronic patent database to contain massive numbers of errors. We have to expect this, if only because of the sheer amount of information involved.   Another reason might be that the economic model of patent data production does not really encourage the national patent offices to maintain high quality electronic patent data. There are millions and millions of patent documents pouring out of government-run institutions (without a profit motive for perfection), and errors are bound to be rampant.

Now, what do I really mean by “massive numbers” of errors? I wish I could come up with a percentage, and there might be a study or two out there that give predictions. But the takeaway point is that there are enough errors to be significant. Enough errors to potentially impact your search. The fact that multiple patent documents were completely missing from the database should be enough to drive that point home.

Rick continued with the following:

I understand from numerous presentations by patent data vendors and my own work in that field, that one aspect of the value added by patent data vendors compared to the USPTO public databases is cleaned up data.  The vendors run routines on each new batch of patent data looking for and programmatically correcting if feasible, all sorts of irregularities.  So, if you have a high value project (and what project is not high value?), keep these facts in mind.

Now, the vendors that immediately come to my mind that are involved in actively manually correcting data are IFI, CAS, and Thomson Reuters (the manually-edited DWPI and DGENE, for example).  I think I have heard that CAS also applies corrections to their incarnation of their INPADOCDB database.  We know that GenomeQuest manually corrects sequence data for some low-quality OCR’ed documents, and I believe I have heard that the World Intellectual Property Organization (WIPO) manually corrects the lowest-quality 2% of the documents they OCR each week.   This is just for starters – please contribute me in the comment section if you’d like to add others.

To drive the point home, Roy Zimmermann of Medtronic added an even more distressing anecdote about a possible source of gaps in the database, generated by difficulties converting paper to electronic data:

Also, recall that when the PTO announced its intentions to throw out it hardcopy classified patents files, the “shoes,”  a group of patent searchers that worked primarily using those manual resources in the PTO, NIPR, did samplings of the shoes compared to the online database contents, found substantial discrepancies, brought a lawsuit to force the PTO to cease discarding the paper copies until it could assure completeness.     As I remember, the discarding was halted for a few weeks, then resumed without any sufficient effort to demonstrate the online contents matched the paper contents.    So while we may be concerned about gaps or omissions in the PTO databases, we shouldn’t be surprised by them.     This is a known problem of long standing!

Another point Roy made is that the USPTO introduced free searchable patent data available only back to 1976 via its website, but it has long since produced a searchable backfile going back even further.  Early in my career I went through a period of obsessively reading PIUG listserv archives, and I remember that there was a huge push at one time to get this database to become available to the public for free.  Although the USPTO website only offers the data back to 1976, we now have the independently-created US full text database offered by Google Patents, and Google also hosts the USPTO data for bulk download.  But even with the complete backfiles, there’s no reason to believe that this collection is really *fully* complete.

As a nice conclusion to the story, the discussion continued over on the PIUG wiki, with PIUG Chair Tony Trippe contacting the always-available Commissioner Stoll for more information.  It seems that Commissioner Stoll was able to get a spotlight on this issue and publicized contact information for Larry Larsson, the point person for USPTO database questions.   This conclusion highlights the growing relationship that PIUG has with PTO leadership, which I think will benefit the entire searching community.

UPDATE: For a further enlightening discussion of the gaps in the USPTO full text database, please see the comment section of this post (click the word “Comments” where it appears at the very end of this post).

Do you have your own anecdotes of missing US patent data?  Also, what data correcting vendors did I miss?  Let me know in the comments!

This post was edited by Intellogist Team member Kristin Whitman.