Dealing with whitespace in data types

tgarnett

How would you approach creating your Data Types when you are running into situation where you
are getting extra white space?

For instance I am creating my address Data type and I am getting a lot of

P. O. Box 4 35

where there is a space in between the 4 and the 3.

RandoCalrisian

I actually made some Grooper objects to help illustrate this problem and solution. These are Grooper 2.6 objects.

This first file (RegEx Currency with possible space or Period - Batch.zip) is a Batch Object containing a sample Page Object I created to illustrate the problem (forcing a period, a space, and a non-space.) It has been OCRed.

The second file (RegEx Currency with Possible Space or Period - DT) is a Folder Object with a Data Type containing Data Formats Illustrating both solutions.

GrooperGuru

There are two schools of thought here. The first is to use text pre-processing on the format/pattern and remove control characters -> spaces. This effectively removes every space character from the document prior to the pattern running. This technique is useful for numbers, but causes issues with multiple word values like what you describe here.

The "better" approach (starting in 2.6) would be to leverage Fuzzy RegEx mode on the format/pattern. The pattern would end up being something like:
P[.] O[.] Box \d{1,4}

As long as the Fuzzy Match percentage is reasonable, this technique should work well and will remove the additional space in the output.

RandoCalrisian

I can add to this, in order to get a better percentage for the FuzzyMatch method, throw in a LookAhead and LookBehind (if possible) to make the string longer, and as a result, make the Fuzzy Confidence higher (since FuzzyRegEx works on the entire pattern including the LookAhead and LookBehind.)

GrooperGuru

RandoCalrisian said:

I can add to this, in order to get a better percentage for the FuzzyMatch method, throw in a LookAhead and LookBehind (if possible) to make the string longer, and as a result, make the Fuzzy Confidence higher (since FuzzyRegEx works on the entire pattern including the LookAhead and LookBehind.)

Definitely agree with you here. Especially when dealing with short values like currency amounts. 3-5 character strings don't give fuzzy percentages enough headroom to be practical.