Sunday, November 2, 2008

Lookup Cache

Lookup transformation supports 3 types of cache.
Full Cache - This brings the reference table into memory - Its good if you have lots of memory. Partial - It adds rows to the cache as they are used - Good if using less memory or the range of rows looked up could be less.
None - Always queries the Database every time - useful in low memory conditions

Users sometimes run into a puzzling situation when they use caching in the lookup. With caching settings set to Full Cache they get no hits - all the rows flow to the error output. However, with caching set to Partial or none, (memory restriction is enabled), things work as expected. The common reason for this is that they are comparing strings which have leading or trailing spaces, for example: “John” with “John “

When caching is Full (memory restriction is off) SSIS builds a hash table for the cache in order to do the comparisons more quickly. The hash values for the strings quoted are, of course, different.
When memory restriction is on, string compares are used and the two quoted strings match.
You can trim incoming datea in the source query or using an expression.
Trim reference data using a SQL query rather then selecting a table in the lookup. SQL queries are better in lookup anyway because you only cache the columns (and rows) you asked for, rather than the entire table.

Lookup is case sensitive, so you may need to force the case for the test to succeed.
Data types must match exactly. The join columns, between the input or source data and the reference data must be of exactly the same type.Lookup does not support the full range of SSIS data types. The following types cannot be used as join columns.

DT_R4
DT_R8
DT_TEXT
DT_NTEXT
DT_IMAGE

No comments: